A look at the otherworldy scope of AI chips

May 24th
Cerebas's dinner plate-sized AI chip. (Cerebas)

This post is sponsored by Brilliant, which offers courses like Introduction to Neural Networks, where students can study the challenges of learning and perception.

Other courses include _Computer Memory, which focuses on memory management, layer by layer, and _Introduction to Linear Algebra, which touches on “just about all modern scientific subjects, including physics, mathematics, computer science, electrical engineering, economics, and aeronautical engineering.”

Sign up for Brilliant here to get 20% off an annual membership today.

Nvidia recently announced its new DGX A100 artificial intelligence chip, which it says is 20 times faster than its predecessor and a lot cheaper to use. There’s a lot to admire about AI chips and a lot that makes them different from graphics cards. Let’s take a look.

First, the cost. As we’ve explored in past explainers, neural networks are extremely expensive to compute. The sheer amount of data they require may become a problem for the discipline in the future.

It stands to reason that the chips used to run them are expensive. Unlike general purpose hardware, AI chips are tailored for neural architectures — that is, they have to be able to run simultaneous, parallel calculations to power the artificial neurons that are designed to fire inside neural thickets like the human brain](/imagenet/)

For an example of this, look at the new A100’s transistor count — it has 54 billion transistors (on-off switches), which is nearly 30 billion more than a top-of-the-line 2080 Ti graphics card The size of the silicon die is another sign of both expense and computing necessity, and the A100 is nearly 100mm larger than the 2080 Ti.

Furthermore, while some machine learning can be done on laptops and edge computers like the Raspberry Pi, real enterprise work is done in the kind of antiseptic server rooms you see in movies. Nvidia’s new chip, despite its reportedly absurd upgrades, should be far, far cheaper than the rest of the market.

From VentureBeat:

For instance, to handle AI training tasks today, one customer needs 600 central processing unit (CPU) systems to handle millions of queries for datacenter applications. That costs $11 million, and it would require 25 racks of servers and 630 kilowatts of power. With Ampere, Nvidia can do the same amount of processing for $1 million, a single server rack, and 28 kilowatts of power.

But what about other companies? There’s Cerebas, IBM, Qualcomm and Google in the US. In South Korea, there’s Samsung. In China, there’s ZTE and Huawei.

Let’s look at Huawei — the company has worked on in-house AI chips since 2018, and announced its Ascend 910 in the fall of 2019. Compared with the just-announced A100, it’s relatively small. The Ascend 910 delivers 512 teraflops, or 512 trillion floating-point operations per second. As a benchmark for computation, this is incredibly impressive. It pales in comparison with Nvidia’s newer A100, though, which delivers 5 petaflops, or 5,000 teraflops, of operation.

Huawei, a far newer entrant into chipmaking and operating under the cloud of the US entity list, has done well, however.

Google made some major advancements of its own in last years of the 2010s with its Tensor processing unit chip design, an alternative to graphics processing units, or GPUs. As it and other tech giants pivoted to AI, the need for cards designed specifically for deep learning and neural networks proliferated. For an already expensive discipline, using cards made for graphics rendering would ultimately cost companies billions of dollars in wasted energy.

Another chip worth looking at is the dinner plate-sized AI processor made by Cerebas, which calls its chip the largest in the world. Cerebas’s flagship chip may be 100 times the size of an average chip design, and according to its CEO, up to 1,000 times faster.

It could be far hotter, as well. And that’s the problem — while others have built giant computer chips in the past, they’re very difficult to maintain.

“The problem is that they have not been able to build one that is commercially feasible,” Rakesh Kumar, a professor at the University of Illinois, told the New York Times.

It hasn’t stopped Cerebas from finding clients, however. One of them is also the first client of Nvidia’s A100 — the US’s Argonne National Laboratory. Argonne plans to use Cerebas’s 64 square inch chip to help advance drug discovery. According to Nvidia, Argonne will use its A100 chips to research Covid-19.