For me, it won't be the agent's office because I don't want to be published by someone else. What's an agent going to do? Negotiate with me on my own behalf? Wouldn't work. Publishers are still approaching by the way, and very confused when I turn them down. Increasingly, with tech nightmares and so forth, I think I'm an idiot to try to do this alone, but I'm going to and that's all there is to it.
Anyway, leaving my own situation aside for a second, regardless of your route to publication, there will be a day when nobody can hold your hand. Maybe it's the first time you receive a rejection. Nobody is in the house. It's just you and the agent/editor/publisher's sub-sub-sub-sub assistant. How do you handle that? Let's, for fun, imagine the electricity has gone out and your phone doesn't work. You can't access any of us. You're on your own. How do you cope when you don't have back-up?
My scenario's easy to work out. I've put myself in a trickier than necessary situation because I so want to publish myself, but for you it might be the opposite. Maybe you struck a deal with an agent, who struck a deal with a publisher. Maybe wine was quaffed and plans were made. Maybe, now, the publisher can't live up to their end of the deal due to the climate. Maybe they're not returning your calls, or you're being handed around to different editors. Maybe the new editor doesn't feel the same way about your work. You're going through the motions. The buzz has buzzed off. Maybe you need to put yourself into a difficult situation. Maybe you need to leave. When you have to make a tough choice, that may not make immediate sense to others, it's only you and yourself who can decide. When you're all alone, with only yourself for back-up, what do you do?