MULTI-MODAL PRE-TRAINING METHOD AND MULTI-MODAL PRE-TRAINING APPARATUS
Number of patents in Portfolio can not be more than 2000
United States of America
Stats
-
N/A
Issued Date -
Nov 14, 2024
app pub date -
May 13, 2022
filing date -
May 13, 2022
priority date (Note) -
Published
status (Latency Note)
Importance
Abstract
The present disclosure provides a multi-modal pre-training method and apparatus. The method includes: sampling a video in a video-text pair to obtain a first video frame sequence; performing word segmentation processing on a text in the video-text pair to obtain a first word segmentation sequence; masking on the first video frame sequence to obtain a second video frame sequence; masking on the first word segmentation sequence to obtain a second word segmentation sequence; encoding the first video frame sequence to obtain a first video feature, and encoding the first word segmentation sequence to obtain a first word segmentation feature; encoding the second video frame sequence to obtain a second video feature, and encoding the second word segmentation sequence to obtain a second word segmentation feature; performing multi-modal pre-training by using the first video feature, the first word segmentation feature, the second video feature and the second word segmentation feature.
First Claim
Family
International Classification(s)
Cited Art Landscape
Patent Citation Ranking
Recipient Email Address
Recipient Email Address
Comment
Recipient Email Address
Add to Portfolio(s)
To add this patent to one, or more, of your portfolios, simply click the add button.
This Patent is in these Portfolios:
Add to additional portfolios:
Last Refreshed On:
Changes done successfully