René Descartes, in the mid-17th century, was the first to connect thought with the engineering of his day – in his case living systems were mechanical automata with plumbing or hydraulics to move ideas and actions.
Now, brains are too often thought of by neuroscientists as little computers with a processing capacity and a hard drive.
The internet, however, is a better metaphor for the brain, according to the book, An Internet in Your Head, by Daniel Graham. Both are places of trying to sort and process information in the most rapid and effective way.
As Graham writes: “In some ways, it is odd that it has taken so long to recognise that flexible, efficient, reliable communication is precisely what both the internet and the brain do.
“The key innovations – such as a system for dividing up messages into chunks of fixed size – have been known for more than half a century. We should not expect that the brain works exactly like the internet. But tricks similar to those used by the internet seem necessary in the brain as well.”
And as well as in making comparisons to the internet, computer scientists are looking more closely at the brain’s tricks to see if insights can be found for the infrastructure underpinning artificial intelligence (AI).
Zaid Kahn, general manager in Microsoft’s cloud hardware infrastructure engineering group, where he leads a team focusing on advanced architecture and engineering efforts for AI, said: “AI currently is in an arms race.
“Because the industry is at a young stage to train large and useful models there is a large chip landscape of companies, such as Graphcore and Nvidia, to process these large models in a dense matrix of parameters.
“To build and scale the infrastructure to train models is not a trivial task to put together at a large scale. It requires enormous power – 300 to 500 Megawatts and this is increasing.
“So I look at this and at some point think efficiency will become important. The brain is 1,000 times faster using 25 watts of power – equivalent to a lightbulb – because humans know five things rather than record every detail. This is the sparsity model and a big opportunity.”
In AI inference (which are the capabilities learned during deep learning training are put to work) and machine learning, sparsity refers to a matrix of numbers that includes many zeros or values that will not significantly impact a calculation.
The goal, as Nvidia noted last year when developing its third-generation Tensor Cores, is to reduce the mounds of matrix multiplication deep learning requires, which shortens the time to good results without losing accuracy.
To this end, Nvidia, other established chip companies and startups, such as SambaNova, Horizon Robotics, GraphCore, Groq, Nuvia (recently acquired by Qualcomm), Cambricon, and Cerebras, have been raising money to tackle inference and training systems.
This, however, is hard to do because the feedback loop from model to measuring accuracy can be lengthy and the gap between hardware and software can cause issues.
Software near the end user or customers – the upper stack – has greater fluidity and can iterate faster than the chips or hardware that become outdated within a year.
This creates tensions for the bit in the middle – the lower stack – where the software code compiler sits and tries to optimise for the different hardware and deliver the output. Ultimately, these compilers are expected to plug in to all the successful chips and hence churn out the code the software applications can use.
AI will then write the software and potentially optimise the compilers.
Jeff Herbst, vice-president of business development at Nvidia and head of Nvidia GPU Ventures, spoke to George Hoyem, managing partner at In-Q-Tel, at the GCV Digital Forum 2021 in end of January: “Modern AI is basically pattern recognition on data, whether it is images or voice.
“Fundamentally what is going on in the world right now is that the traditional model of how computers are programmed has been turned on its head.”
As Rashmi Gopinath, general partner at venture capital firm B Capital Group said, training and inference time, energy consumption and memory usage of AI is catching up with the dynamism of open source software and potential of quantum and classical hardware. Gopinath will be speaking at and a speaker at next month’s Software for AI Optimization Summit.
And once the stack is being optimised for performance by the algorithms having clarity on the data inputs and model outputs becomes more of a focus.
The revolutions of electricity, semiconductors and the internet has transformed society and business and created the conditions for data and information to be created and shared.
Now it is a race to when convergence between humans and AI or the singularity can happen, which some AI scientists, such as Jürgen Schmidhuber, speaking at Nvidia GTC 2021 last month, expect by 2040.
By then the next metaphor for the brain and computers will have been developed by algorithms, such as GPT-3.
Quantum’s opportunity
Traditional computers are based on ones and zeroes, or binary. These binary positions are usually transistors, which are on or off with electricity.
In quantum computing, the transistors are the state of the electrons, whether they are spinning one way or another often as directed by a laser. This allows them to be much smaller because they are less than the size of a whole atom and effectively each one trapped in a quantum dot, as Nick McClure, head of data science at Hitch, noted in his review of the industry.
Quantum dots can be in multiple states rather than just on or off, which allows applications to be developed that tackle complex calculations, such as market risk in financial services.
Last month, investment bank Goldman Sachs and US-based startup QC Ware said they had found a way to use quantum computing for the Monte Carlo algorithm that is used to evaluate risk and simulate prices for a variety of financial instruments. Researchers sacrificed a 10th of the speed to produce shallow Monte Carlo algorithms capable of being run in quantum equipment under development.
VC-startup alliance
Nvidia has launched an Inception VC Alliance between its startup acceleration platform and several venture capital firms to support the growth of artificial, data science and high-performance computing startups.
The VCs are Acrew, IQT, Madrona, Mayfield, NEA, OurCrowd, Pitango and Vanedge.
The programme offers access to high-profile events, visibility into top startups raising capital, and access to resources for portfolio companies.
Membership is free but there is a short application process.