When I started watching anime I had a hard time with Japanese audio. Back then I got my hands on some great shows with above average, to great English voice acting: Cowboy Bebop, Kenshin series, Ghost in the Shell (movie), Chobits,... And I really liked them in English. However, as I dunk more into anime, I watched more and more titles in Japanese, and I enjoyed them immensely; to the point that when I rewatched the above shows, I did so in Japanese audio, and I couldn't stand the English voice acting anymore!
Don't get me wrong, some titles got exceptional dubs (Bebop and GitS immediately come to mind), however, I think that jokes and innuendos are enjoyed best when heard in their original language, and dubbing kills some of their effects. This becomes more apparent as you continue watching shows in their original voice acting, as you become familiar with them and you start "getting" the jokes without resorting to the subtitles.
Personally, I don't watch any anime in English anymore, so I always prefer a single-audio (Japanese) releases

The same goes for the Japanese games, esp. RPGs

I played some of my favorite titles on PS2 as their undub version (Japanese audio with Eng sub, playable only with hacked PS2 or an emulator) which is quite amazing, and those titles which allow you to choose the audio language always win my love