Here is one of the posts I really like. It kind of gives a good concise example of his approach: http://www.kaushik.net/avinash/2007/11/blog-metrics-six-recommendations-for-measuring-your-success.html

He also has a book.

A good friend of mine who does this kind of optimization for a living at Google said:

“I put the laptop on my desk, load the web page, stand 10 feet back….and if I can’t tell what I’m supposed to do, I’ve failed.”

Here is something along those lines that might help. It’s basically just a front end for the mechanical turk that does split testing. The idea easy, non-scientific market research. Basically an online equivalent of asking a bunch of people in a cafe a question.

You might be able to use it for a similar purpose to that 7 second test, but it’s not built for that.

http://pickfu.com/

I’m not sure how to answer that. I suppose the options are: pay, learn or do without.

pay – You can always hire someone. If your problem is making something look nice, you can get design services at every price range. You can have pages mocked up in photo shop at very low prices. You can have themes for popular CMSs applied fairly cheaply too. If what you want is help with usability, help can be pretty pricey.

Learning by doing is also always an option. If you are interested in anaylitics & usability, I suggest you start reading this blog: http://www.kaushik.net/avinash/ . Improve by making small changes. Split test, etc. Some people are brilliant designers, & they put all their faith in design. Some people love analytics (read that blog), & they want stats to tell them exactly what to do. Some people came to this field from a software development/usability angle. Everyone ends up learning from a different angle, so it is hard to tell you exactly how to go about learning.

Doing without is not necessarily a terrible option. Google suck at design. They are great at software. They actually do without design as much as they can. They compensate with other stuff like speed & simplicity to make things usable. They can do this by not trying to design. Everything is very simple: Google Reader, Search, Gmail, docs… It is all substantially simpler then most anything else. They just avoid designing.

*Sometimes it doesn’t work. Noone wants a Google Sites site as their company website. Not even as a personal homepage. The design sucks.

That’s my 2 cents.

The second issue is that it really needs to be completed by people unfamiliar with your site, and they’re not going to be the easiest target audience to reach, unless you make it some sort of collaborative thing – ie. you have to review 10 other sites before you can submit your own, or something like that.