You still need to learn it the same way anybody else would. Even without speaking, if you don't incorporate listening as part of your studies, you'll be making life much harder for yourself. Speaking offers a lot of benefits for overall acquisition as well.
I'd bet that your mind will change about speaking as you progress anyway. If you wanted to use it to get an interesting job, nobody is going to hire you if you can't even say hello. If you go to a restaurant, you'll be able to see what you want and not be able to order it. I could give a lot more examples, and I think a lot of people (including myself) don't like the idea at first, but hopefully you'll rethink this at some point.
Anyway, to get a jumpstart on reading and writing try getting a copy of Heisig's Remembering the Kanji, and downloading the flashcard program Anki.
Agreed on all facets. It's really hard to understand Japanese without the spoken part of it, especially because a SIGNIFICANT portion has to do with context, so it's hard to understand this particular language if you don't understand the culture behind it.