As Exophase said, low level is of limited use. Acceleration wise all you have is a blitter (which takes a fair bit of setting up - small blits can actually take longer to set up than what you gain by using the blitter's acceleration) and a display scaler (doesn't actually alter the framebuffer, just how the screen displays it - takes no time to zoom, but you don't have access to sub-pixels, plus no filtering).
Also the framebuffer is in non-cached memory. You can alter the cache settings to allow the CPU to cache it, but you'll need to explicitly flush the cache as the display hardware cannot see the CPU's cache.
Only do it if you really need the speed gain from assembler. Make sure that the drawing routines are causing any slowdown before coding them in assembler.
SDL wil give you direct access to the framebuffer if you want it, that way SDL handles all the set-up for you - as long as you use my version of SDL (also in Open2X, it uses the blitter for blitting / filling, and easy [display]zooming), GPH's is a full software version - the framebuffer is in cached memory (fast user access) but copied to non-cached memory on SDL_Flip (required for display hardware - and take ages, plus NO hardware acceleration / scaling).
Note:
My version of SDL isn't 100% compatable, in most cases it is but a few things don't work as expected (like the cursor pointer needing to be switched off AFTER setting the screen up whereas in normal SDL you can switch it off before), there may be other issues in paletted modes (8-bit and less) as they haven't had as much (read near-zero) testing and I haven't touched the code for a long time, but if you use it let me know if you hit any problems.