The NSA has access to their own chip fabrication facilities. I do not know if they own their own plant, or just have secure fab space at some other company's plant.

So they could have easily fabbed something like this, or a tuned architecture specifically designed for the purpose.

NSA does not own a foundry anymore. Modern top-of-the-line foundries cost too much, even for the NSA.

Instead, they participate in a program to partner with domestic companies to manufacture their chips: http://trustedfoundryprogram.org/

It's very probable that they have and if so, it's almost certain that it would involve specialized hardware implementations (ASIC, FPGA, whatever) rather than commodity graphics hardware which is burdened with expensive and useless stuff like onboard memory and would be power inefficient

considering that the entire purpose of NSA in the first place is to provide SIGINT and encrypt or decrypt signals, it's almost a given that they're trying to the best of their ability to crack stuff.

Not sure

Money buys more commodity hardware faster than the time/money used to develop a chip

It's not hard to make tens, or maybe even hundreds of GPUs beat a specialized chip except for very specific things

And even for something specialized it's probably easier to use an FPGA