Skip to main content

How knowledge distillation compresses neural networks


If you’ve ever used a neural network to solve a complex problem, you know they can be enormous in size, containing millions of parameters. For instance, the famous BERT model has about ~110 million. To illustrate the point, this is the number of parameters for the most common architectures in (natural language processing) NLP, as summarized in the recent State of AI Report 2020 by Nathan Benaich and Ian Hogarth. You can see this below: In Kaggle competitions, the winner models are often ensembles, composed of several predictors. Although they can beat simple models by a large margin in terms of…

This story continues at The Next Web

from The Next Web https://ift.tt/3miEaGu
Labels:

Comments

Post a Comment

Popular posts from this blog

TNW Podcast: Boris comes over to co-host; Slack’s Cal Henderson talks European tech

 Welcome to the new episode of the TNW Podcast — the show where we discuss the latest developments in the European technology ecosystem and feature interviews with some of the most interesting people in the industry. In today’s episode, Andrii is joined by Boris Veldhuijzen van Zanten, co-founder, member of the board, and former CEO of TNW. The topics discussed include the jobs created by Dutch startups, giant state funding for energy projects, translations of the word ‘computer’, and a bunch of other things in between. In the interview section, we’re featuring a conversation with Cal Henderson, co-founder and… This story continues at The Next Web from The Next Web https://ift.tt/jUgcNFD
Post a Comment
Read more

How to design human-machine interfaces for vehicles of the future 

A human-machine interface (HMI) is an interface that allows us to interact with a digital system. No matter what HMI we design, we need to enable users to take advantage of all that a system offers.  For almost two decades, the personal computer was the first thing that came to mind when we heard about digital HMI. But the situation changed, and today HMI is an integral part of many devices we use daily — mobile phones, smartwatches, IoT devices, and even cars. Car HMI design is a relatively new field with specific challenges. My team has experience designing for… This story continues at The Next Web from The Next Web https://ift.tt/3m7s0m6
Post a Comment
Read more

2024’s most important travel tech trends

Guest post by Singapore Tourism Board As the tourism industry continues its post-pandemic recovery in an increasingly uncertain travel market, a question that has dawned upon us is — what’s next? Naturally, many of us are looking towards technology for more opportunities. In 2023, we saw the generative AI boom and further growth of emerging technologies like extended reality (XR) and 5G. While these advances are already shaping the travel sector, there is much more to come as more use cases emerge while the technology further matures. Looking forward into 2024, we can expect teams to do more with less, streamline their… This story continues at The Next Web from The Next Web https://ift.tt/seCScOH
Post a Comment
Read more