To answer the original question: the answer is no, you don't need any closed drivers. Of course that means you don't have wifi/bt (except maybe via USB if you can find one that has an open driver) and you cannot use the GPU.
Lots of the 3D stuff on the Pandora (e.g. Playstation emulation with PCSX ReARMed) doesn't use the GPU anyway, because optimized NEON code seems to be faster or more reliable.