Opera Dragonfly goes in that direction. It's open-sourced (http://bitbucket.org/scope/dragonfly-stp-1/) and uses documented protocol for communication with browser core:
http://dragonfly.opera.com/app/scope-interface/
Recently Dragonfly has been made compatible with Chrome, debugging Opera remotely:
http://twitpic.com/30ul4b