Ulam has much the same sentiment as the man in the last two panels. Rota quotes Ulam (in his book Indiscrete Thoughts, p. 58) as saying:
What makes you so sure that mathematical logic corresponds to the way we think? Look at that bridge over there. It was built following logical principles. Suppose that a contradiction were to be found in set theory. Do you honestly believe that the bridge might then fall down?
My personal view is that the sentiment above is wrong: if 1 and 1 don’t equal 2, then the bridge couldn’t be built in the first place, and civilisation couldn’t get off the ground from the start, since counting couldn’t work.
What Ulam and the man above are suggesting is that it’s possible for mathematics as we know it to go wrong, and for the world to remain the same. But I cannot agree with that assumption: mathematical truths are (to me) paradigms of necessary truths.
To see which side you’re on, try this thought experiment (not original to me):
Imagine that in this world, whenever people add 2 objects and 2 objects together, a malicious demon always adds 1 object. So two oranges stacked with two oranges become five oranges, and so on. Do you think our mathematical calculations would come out any different? Would we conclude that 2+2=5, or would we still have our normal law of 2+2=4?
If one were to find evidence indicating 1+1 was not =2, that would not mean that the world is wrong but our way of looking at it and describing it. In other words, if you come to believe 1+1 is not =2, your problem is not reality but your idea of ‘one’, ‘and’/’plus’, ‘is’/’equals’, and ‘two’.