A fortress is a heavy symbolic idea of keeping people out and in for safety reasons. Things trying to attack -- no problem, things trying to escape -- no problem. A fortress is there to keep things, whether you like it or not.
This is why, "Silence is a strenuous language / but we have chosen it." There's a sense of tension between the "we" and if something is said then it could open a delicate situation; however, what this leads to is an undefined language where actions are taken to overblown proportions, "A shut door, a shrug, / stone upon stone."
The image of a the stone reoccur again in the next stanza. This image of something steady and solid, but silent is given context of, "The stones have a history / They were pulled from the rubble / of an earlier weekend." Note here that "earlier weekend" pretends to be a specific time frame, but is rather due to the usage of earlier. Yes the stones come from a weekend somewhere in the past -- sometime earlier.
The next stanza brings the tension ahead:
After each skirmish we retreatThe play of tactile imagery of warm and silence parallels a push and pull of a relationship. The warmth coming after a skirmish (an understanding to bring closeness) and still it's cold -- which feels like the overall feel of the "we." When the stanza ends with a proclamation it sounds like a war cry and exasperation all in one.
to warm ourselves at a silence.
And still it's cold
Let the cold be our comfort!
Now the last stanza I have is different from the last stanza from the website. The white space in line two and three does make a difference "We live in fear. / A single word." Since it feels like the single word is what the speaker is waiting for to crumble the fortress. But with the stanza left aligned there lacks the tension of wanting something to topple and not at the same time.