The notion that Mathematics is a language is held by many mathematicians and is being expressed on frequent occasions. Below, I am going to collect quotes I occasionally come across that uphold that point.
My own attitude, which I share with many of my colleagues, is simply that mathematics is a language. Like English, or Latin, or Chinese, there are certain concepts for which mathematics is particularly well suited: it would be as foolish to attempt to write a love poem in the language of mathematics as to prove the Fundamental Theorem of Algebra using the English language.
Mathematics is pure language - the language of science. It is unique among languages in its ability to provide precise expression for every thought or concept that can be formulated in its terms. (In a spoken language, there exist words, like "happiness", that defy definition.) It is also an art - the most intellectual and classical of the arts.