I don't necessarily go through these entire checklists every time I want to download something, and sometimes things like fansub group fanboyism will take precedence. But imo these are good things to look for.
For fansubs, I look for:
* A resolution I can comfortably play (576p down to 360p or so, though I will watch 240p if need be)
* Subtitle styles
designed to be readable instead of "to be unobtrusive" (= background-blending CAPTCHA-subs) or "to fit the anime" (usually means esoteric and stupid fonts). Hardsubs are okay if they're readable; softsubs should be no less readable, especially if numerous dialogue styles are used. Having to change 12 different color-coded softsub styles because they're too small or too low in the image makes a softsubbed release effectively hardsubbed.
* Songs translated with English lyrics at the minimum, though romaji with fabulous hardsubbed, simple softsubbed, or no karaoke effects is a nice bonus. Kanji doesn't need to be there, but I don't mind if they're present.
* All relevant and amusing text decently typeset. Hardsubbed AFX typsetting is fine too.
* Localization: not overly-localized to the extent of 1990s official subs (like yen to dollars, riceballs to doughnuts, that kind of stuff), but also free of pointless undertranslation (like SS-Eclipse's Shana releases ~de arimasu) and weeaboo elements. No "keikaku means plan" TL notes -- if you have to put a note saying "Japanese Word A means English Word B," you should probably just put English Word B. I don't care if words/phrases like itadakimasu, ojamashimasu, ittekimasu, gochisousamadesu, otsukaresama, etc. are sacred cultural relics. They're translatable, so they should be translated. Honorifics, Eastern name order, and other Japanese "flavor" elements are okay. What I mainly look for are "liberal" subs that flow well in English, to the point where the subtitle script is a solid piece of English standalone reading material. Creativity, variety, and context-sensitive writing that fits the characters is the key.
* Accurate translations, with as few English technical errors (spelling/grammar/word use) errors as possible.
* Audio/video quality isn't too important, since it's dependent on the quality of the raws or transport stream source. But obvious macroblocking and artifacting that indicate bitrate starvation are never good.
* Subtitle timing that employs sufficient amounts of lead-in and lead-out, avoids distracting errors like "flash" (short gaps between consecutive lines), "flicker" (lines start just after or end just before a scene change) and "scene-bleeds" (lines start just before or end just after scene changes). Subtitles should stay on the screen neither too short nor too long. Too short, and you get a bunch of fragmented sentences without a lot of time to read them. Too long, and you probably give away too much information too early, which disrupts the dramatic flow. Staying within 2-5 seconds where feasible is a good guideline.
* Some "trolling" is fine if it's in a non-serious anime (Kampfer, B Gata H Kei, Needless, Baka & Test) or an unintentional comedy (Code Geass, Umineko, Star Driver).
For "re-subs" that apply fanmade subs from TV releases to DVD or Blu-Ray encodes, I mainly look for:
* Video quality that's a significant improvement over the TV rips, or at least offers enhancements like decensoring, animation fixes, and new scenes.
* Subtitles that provide an experience no worse than what the TV-rips did -- if a song or relevant sign was translated in the TV-rip, it should also be translated in the BD-rip. "But the group hardsubbed their kara/signs, it's too much work!" isn't an excuse, as I routinely release DVD-rips of shows that were 100% hardsubbed. And doing things like leaving songs out because the rip group/individual thinks they're pointless is just pushing their own agenda and value judgments on viewers. Fail.
* Subtitles that do not introduce mistakes that weren't in the TV-rips, and preferably fix layman-fixable mistakes, i.e. things you don't have to be a translator or talented editor to fix. Running Aegisub spellcheck and Timing Postprocessor is *not* that time-consuming.
For rips of R1 DVD/BluRay, and general encoding preferences:
* If English audio is present in the source, it should be present in the release. In this day and age of relatively cheap huge HDDs, when people embrace FLACebo audio and ridiculous bloated upscales, I find it hard to believe that the disk space taken up by lossy English audio still matters, I suppose it matters if you're burning on CD-Rs, which is why I call dub-removing encodes "CDR-scrimping" releases. Rips of Japanese releases that don't add in the English audio are acceptable.
* Unedited R1 subtitles are generally better than poorly-edited fansubs, but ripped subs should deliver subs that are no worse than the original DVD subs. Introducing mistakes via careless OCR or failed attempts at enhancing/editing = bad. Using the original DVD imagesubs in .mkv = neutral. Getting actual talented editors, typesetters, retimers and QCers to enhance the DVD subs = good.
* Video encodes should be as big as they need to be to preserve major details and prevent blocking, without pointlessly bloating the files to preserve irrelevant details. That's just diminishing returns. Encoding to varying file sizes depending on what's in the episode is preferable to encoding every episode to a constant file size, like in the CD-R days. Constant file size means that subdued episodes with lots of "talking heads" scenes will be bloated, while episodes with tons of action or "bitrate hog" sequences like rain/snow/grainy flashbacks will be starved.
* Audio shouldn't be bitrate-starved, either. Either use original AC3 audio from DVD, or transcode to AAC or vorbis at sufficiently high bitrates. I won't *avoid* a release with FLACebo audio, but I sure will look for alternatives.