Leveraging Redundancy in Attention with Reuse Transformers
Number of patents in Portfolio can not be more than 2000
United States of America Patent
Stats
-
N/A
Issued Date -
Apr 13, 2023
app pub date -
Oct 5, 2022
filing date -
Oct 5, 2021
priority date (Note) -
Published
status (Latency Note)
Importance
Abstract
Provided are systems and methods that improve the computational efficiency of Transformers or other attention-based neural networks or machine learning models by re-using a number of attention scores between layers and/or heads of the model. To reduce the computational cost of self-attention-based models while achieving comparable or even superior results, example aspects of the present disclosure propose a novel architecture that reuses attention scores computed in one layer in one or multiple subsequent layers.
First Claim
Family
International Classification(s)
Cited Art Landscape
Patent Citation Ranking
Recipient Email Address
Recipient Email Address
Comment
Recipient Email Address
Add to Portfolio(s)
To add this patent to one, or more, of your portfolios, simply click the add button.
This Patent is in these Portfolios:
Add to additional portfolios:
Last Refreshed On:
Changes done successfully