Why would a non-believer in Judaism mourn the fall of the Temple? It was merely the destruction of a building which's main purpose was to kill animals and serve the people's collective imagination.
While that is strictly true, I don't look at the 9th of Av or the days leading up to it as simply a time which commemorates the simple physical destruction of this one building, but I look to the wider scope of what went on and how the events affected Jewish history. The invasion of the Babylonians stole the last vestiges of independence from Judea, forced thousands of Jews into exile, and mercilessly killed thousands of Jews and looted the nation. A terrible tragedy worthy of remembrance even without the religious reasons.
The time of the destruction of the Second Temple was arguably worse. While the Jews didn't have independence at the time, so they couldn't lose it, the crushing Roman offensive killed hundreds of thousands of Jews, razed whole cities like Jerusalem, and removed Jews as a real political authority in the land until 1948. And not only that, but the Jewish Zealots were waging a civil war on the moderates and went about killing the Jewish leaders who were not as extreme as they were. The Zealots had burned the food stockpiled in Jerusalem in preparation for the Roman siege with the view that such resources were a security that would enable people to duck the fight. The Zealots intentionally starved the denizens of Jerusalem to force them to revolt against the Romans.
The Rabbis have said it was baseless hatred among Jews that caused the churban. In part, I would say. Although the Romans would have won even without the Zealots' war, a lot of death and suffering could have been avoided had they refrained.
We don't need to mourn the loss of a building or the supposed metaphysical disfavor the Jewish people have fallen under. There were real awful tragedies that occurred. Besides for the Holocaust, these events were the worst calamities ever to afflict the Jewish people.
So I don't listen to music, I don't shower with warm water, I don't eat meat or drink wine, I don't cut my hair or shave my beard, I keep a more somber attitude than I would usually because these terrible events were awful in their own rights without the religious projections added onto them. I mourn, not for the Temple, but for my ancestors' loss of life, liberty, and property. I commemorate what happened because they need to be and ought to be.
Too many Jews who will fast this coming Sunday will do it for a microcosm of the actual events. Even if you are religious, the Temple being destroyed is a minor event in the huge horror of what transpired. Widen your focus and you will find real reason to lament.