Well this is mildly disconcerting: ccache can make gcc crash. Now, I am the first person to extol the virtues of ccache; the tool dramatically decreases the amount of time that the FATE farm has to spend rebuilding FFmpeg. But the crash is quite reproducible.
If you have kept up with FATE for any length of time, you know that there is a chronic problem with compiling FFmpeg using SVN versions of gcc on PowerPC — in particular, the build phase tends to crash with an internal compiler error. Last night, I upgraded the gcc-SVN compilers on all the FATE machines. Tonight, I was determined to sit down and finally go through all the steps to submit the highest quality gcc bug report I could. I thought it would be prudent to report the full command line without the ccache command. Wouldn’t you know, the problem file compiles fine. From there, it takes over 20 minutes instead under 5 to compile the whole tree. Running the FATE test suite also passes with flying colors.
So there’s a bug here. Where do you think the problem lies? ccache or gcc? I have a bad feeling that this is the kind of problem both camps are likely to blame on the other. All I know is that the following command:
$ /usr/bin/ccache /usr/local/gcc-138544-20080801/bin/gcc -DHAVE_AV_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_ISOC9X_SOURCE -I. -I”/home/melanson/ffmpeg/ffmpeg-main” -fomit-frame-pointer -maltivec -mabi=altivec -Wdeclaration-after-statement -Wall -Wno-switch -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wno-pointer-sign -Wcast-qual -Wwrite-strings -Wtype-limits -O3 -fno-math-errno -fno-signed-zeros -fPIC -DPIC -c -o libavcodec/imc.o /home/melanson/ffmpeg/ffmpeg-main/libavcodec/imc.c
yields the output:
/home/melanson/ffmpeg/ffmpeg-main/libavcodec/imc.c: In function ‘imc_decode_init’:
/home/melanson/ffmpeg/ffmpeg-main/libavcodec/imc.c:146: warning: assignment from incompatible pointer type
/home/melanson/ffmpeg/ffmpeg-main/libavcodec/imc.c: In function ‘imc_decode_frame’:
/home/melanson/ffmpeg/ffmpeg-main/libavcodec/imc.c:808: internal compiler error: in tree_node_structure, at tree.c:2412
Please submit a full bug report,
with preprocessed source if appropriate.
See http://gcc.gnu.org/bugs.html for instructions.
Removing the prepended /usr/bin/ccache command makes the internal compiler error go away. For the time being, I have removed ccache, but only from the problematic FATE configuration. The PowerPC machine will not be nearly as efficient at performing FATE work, but at least it will keep abreast of gcc-SVN changes.
Update, 2008-08-05: Great news: Recent revisions of FFmpeg have been causing the gcc-SVN copy for PPC crash, and without ccache. It looks like I will be able to pin this on gcc after all.
Followup:
Have you tried cleaning up the cache and then using ccache again? ccache takes a copy of the preprocessed sources, maybe there is something in the sources preprocessed with an earlier version of GCC that makes it fail.
In general, you should only use ccache with the same exact compiler version, I actually thought ccache prevented you from mixing and matching different compilers, if it doesn’t it should be fixed :/
Cache cleared; no difference. This is consistent with my understanding of ccache’s operation: ccache uses both the preprocessed source file and the full compiler command line when determining whether the compiled object output is already cached. This is one reason that I always compile SVN versions of gcc with unique path names (e.g., /usr/local/gcc-138544-20080801).
BTW, I found your ccache blog post while researching this problem. You are correct about all of the myths (true, when one understands how ccache operates, it makes absolutely no sense to make ccache part of Gentoo’s make.conf options). But your stated reasons are not all correct. ccache’s man page describes the uniqueness that goes into the cache determination. Thus, altering the the CFLAGS as described in your myth #2 will simply result in cache misses for every single file. Actually, I just realized that part of the uniqueness equation involves size of modification times of the compiler binaries. So keeping the same filenames throughout FATE’s gcc-SVN upgrades would still trigger proper cache misses.
Also, ccache uses MD4 instead of MD5. I would have posted all of this on the relevant blog post but comments were closed. :-)
Ehh, I should probably extend the time comments are open, I noticed lately one month starts to feel too little :) Especially considering that the GMN publishes the link to the post more than a month after I published it :P (changed!)
For what concerns myth #2, although I didn’t make it too explicit, I did say that it doesn’t work as a cache when changing flags :) What I said is that _if_ it was to answer from the cache it would break the setup ;) As ccache is not breaking the setup there’s just one choice available :)
And duh for MD4/MD5… I didn’t even check that as I expected it to be MD5… why should they use MD4 at all? o_O Speed?
At any rate it’s pretty funky, can you get the preprocessed output that ccache is trying to compile and pass it manually through the compiler?
About MD4/MD5, I don’t know for sure but I suspect that MD4 is faster than MD5. MD5 is cryptographically stronger than MD4, but cryptographic security is not a requirement for this scenario.
I wonder about the speed. Mostly because the MD5 algorithm is so much more used nowadays and it’s most likely much more optimised. But you’re the benchmark man, I’ll leave it up to you to see if you can find a way to state that ;)