从没有任务标签的数据中学习表征、聚类或异常分数，随后进行领域验证
Loading technique guide…
模式： 借助自监督学习目标,利用无标注数据将模型适配到新领域
原因： 利用大量无标注数据学习领域专属模式,免去昂贵的标注需求
关键洞察： 自监督的前置任务从数据结构中产生监督信号,从而实现有效的领域适配
通过这些精选资源加深理解
Self-Supervised Learning: Generative or Contrastive (Liu et al., 2021)
A Simple Framework for Contrastive Learning of Visual Representations (Chen et al., 2020)
Momentum Contrast for Unsupervised Visual Representation Learning (He et al., 2020)
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (Grill et al., 2020)
BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018)
RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (He et al., 2020)
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (Clark et al., 2020)
SimCLR: A Simple Framework for Contrastive Learning (Chen et al., 2020)
SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (Caron et al., 2020)
SimCSE: Simple Contrastive Learning of Sentence Embeddings (Gao et al., 2021)
SupCon: Supervised Contrastive Learning (Khosla et al., 2020)
BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018)
SpanBERT: Improving Pre-training by Representing and Predicting Spans (Joshi et al., 2020)
ERNIE: Enhanced Language Representation with Informative Entities (Zhang et al., 2019)
ALBERT: A Lite BERT for Self-supervised Learning (Lan et al., 2019)
CLIP: Learning Transferable Visual Representations from Natural Language Supervision (Radford et al., 2021)
ALIGN: Scaling Up Visual and Vision-Language Representation Learning (Jia et al., 2021)
Florence: A New Foundation Model for Computer Vision (Yuan et al., 2021)
DALLE-2: Hierarchical Text-Conditional Image Generation (Ramesh et al., 2022)