Pardon my ignorance but have you consider something like http://www.andyroid.net/. It seems a linux port is due this month. It isn't open source but it is free.
Don't know if anyone has brought up this but their is a gaming device being built with android. http://www.hardkernel.com/
Unless I'm wrong you have the right to use their code. That would be some what of a good starting point I assume.