Transformer-based neural network including a mask attention network

Number of patents in Portfolio can not be more than 2000

United States of America

PATENT NO 12260338
APP PUB NO 20220067533A1
SERIAL NO

17005067

Stats

ATTORNEY / AGENT: (SPONSORED)

Importance

Loading Importance Indicators... loading....

Abstract

See full text

A transformer-based neural network includes at least one mask attention network (MAN). The MAN computes an original attention data structure that expresses influence between pairs of data items in a sequence of data items. The MAN then modifies the original data structure by mask values in a mask data structure, to produce a modified attention data structure. Compared to the original attention data structure, the modified attention data structure better accounts for the influence of neighboring data items in the sequence of data items, given a particular data item under consideration. The mask data structure used by the MAN can have static and/or machine-trained mask values. In one implementation, the transformer-based neural network includes at least one MAN in combination with at least one other attention network that does not use a mask data structure, and at least one feed-forward neural network.

Loading the Abstract Image... loading....

First Claim

See full text

Family

Loading Family data... loading....

Patent Owner(s)

Patent OwnerAddress
MICROSOFT TECHNOLOGY LICENSING LLCONE MICROSOFT WAY REDMOND WA 98052

International Classification(s)

  • [Classification Symbol]
  • [Patents Count]

Inventor(s)

Inventor Name Address # of filed Patents Total Citations
Duan, Nan Beijing, CN 29 111
Gong, Yeyun Beijing, CN 9 15
Jiao, Jian Bellevue, US 58 89
Zhang, Ruofei Mountain View, US 59 1052
Zhou, Ming Beijing, CN 292 5343

Cited Art Landscape

Load Citation

Patent Citation Ranking

Forward Cite Landscape

Load Citation

Maintenance Fees

Fee Large entity fee small entity fee micro entity fee due date
3.5 Year Payment $1600.00 $800.00 $400.00 Sep 25, 2028
7.5 Year Payment $3600.00 $1800.00 $900.00 Sep 25, 2032
11.5 Year Payment $7400.00 $3700.00 $1850.00 Sep 25, 2036
Fee Large entity fee small entity fee micro entity fee
Surcharge - 3.5 year - Late payment within 6 months $160.00 $80.00 $40.00
Surcharge - 7.5 year - Late payment within 6 months $160.00 $80.00 $40.00
Surcharge - 11.5 year - Late payment within 6 months $160.00 $80.00 $40.00
Surcharge after expiration - Late payment is unavoidable $700.00 $350.00 $175.00
Surcharge after expiration - Late payment is unintentional $1,640.00 $820.00 $410.00