How does your manager figure you what you are supposed to work on? What information or ability do they have that you don't? Why can't you do what they do. Ultimately there is only one thing that is missing...responsibility. You aren't responsible. Your manager is. But there isn't any reason you can't be.