https://arolstar52-zhtest.hf.space/index.php?action=history&feed=atom&title=BERTBERT - 版本历史2026-07-14T03:29:42Z本wiki上该页面的版本历史MediaWiki 1.43.9https://arolstar52-zhtest.hf.space/index.php?title=BERT&diff=3535105&oldid=previmported>Kiyoteru Awaji：/* 外部链接 */2025-11-03T18:02:54Z<p><span class="autocomment">外部链接</span></p>
<p><b>新页面</b></p><div>{{NoteTA|G1=IT<br />
|1=zh-cn:斯坦福; zh-sg:斯坦福; zh-tw:史丹佛; zh-hk:史丹福; zh-mo:史丹福;<br />
}}<br />
'''基于变换器的双向编码器表示技术'''（{{langx|en|Bidirectional Encoder Representations from Transformers}}，'''BERT'''）是用于[[自然语言处理]]（NLP）的预训练技术，由[[Google]]提出。<ref name=":0">{{cite arxiv |last1=Devlin |first1=Jacob |last2=Chang |first2=Ming-Wei |last3=Lee |first3=Kenton |last4=Toutanova |first4=Kristina |title=BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding |date=2018-10-11 |eprint=1810.04805v2|class=cs.CL }}</ref><ref>{{Cite web |url=http://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html |title=Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing|website=Google AI Blog |language=en |access-date=2019-11-27 |archive-date=2021-01-13 |archive-url=https://web.archive.org/web/20210113211449/https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html |dead-url=no}}</ref>2018年，雅各布·德夫林和同事创建并发布了BERT。Google正在利用BERT来更好地理解用户搜索语句的语义。<ref>{{Cite web |url=https://blog.google/products/search/search-language-understanding-bert/ |title=Understanding searches better than ever before |date=2019-10-25 |website=Google |language=en |access-date=2019-11-27 |archive-date=2021-01-27 |archive-url=https://web.archive.org/web/20210127042834/https://www.blog.google/products/search/search-language-understanding-bert/ |dead-url=no}}</ref>2020年的一项文献调查得出结论：「在一年多一点的时间里，BERT已经成为NLP实验中无处不在的基线」，算上分析和改进模型的研究出版物超过150篇。<ref>{{Cite journal |last=Rogers |first=Anna |last2=Kovaleva |first2=Olga |last3=Rumshisky |first3=Anna |date=2020 |title=A Primer in BERTology: What We Know About How BERT Works |url=https://aclanthology.org/2020.tacl-1.54 |journal=Transactions of the Association for Computational Linguistics |volume=8 |pages=842–866 |doi=10.1162/tacl_a_00349 |access-date=2021-11-24 |archive-date=2022-04-03 |archive-url=https://web.archive.org/web/20220403103310/https://aclanthology.org/2020.tacl-1.54/}}</ref><br />
<br />
最初的英语BERT发布时提供两种类型的预训练模型<ref name=":0"/>：（1）BERT<sub>BASE</sub>模型，一个12层，768维，12个[[注意力机制|自注意头]]（self attention head），110M参数的神经网络结构；（2）BERT<sub>LARGE</sub>模型，一个24层，1024维，16个自注意头，340M参数的神经网络结构。两者的训练语料都是[[BookCorpus]]<ref>{{cite web |last1=Zhu |first1=Yukun |last2=Kiros |first2=Ryan |last3=Zemel |first3=Rich |last4=Salakhutdinov |first4=Ruslan |last5=Urtasun |first5=Raquel |last6=Torralba |first6=Antonio |last7=Fidler |first7=Sanja |date=2015 |title=Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books |pages=19–27 |class=cs.CV |eprint=1506.06724}}</ref>以及[[英語維基百科]]语料，单词量分别是8億以及25億。<ref>{{cite arxiv |last=Annamoradnejad |first=Issa |date=2020-04-27 |title=ColBERT: Using BERT Sentence Embedding for Humor Detection |class=cs.CL |eprint=2004.12765}}</ref><br />
<br />
== 结构 ==<br />
BERT的核心部分是一个[[Transformer模型]]，其中编码层数和自注意力头数量可变。结构与Vaswani等人(2017)<ref name="vaswani">{{cite arXiv|last1=Polosukhin|first1=Illia|last2=Kaiser|first2=Lukasz|last3=Gomez|first3=Aidan N.|last4=Jones|first4=Llion|last5=Uszkoreit|first5=Jakob|last6=Parmar|first6=Niki|last7=Shazeer|first7=Noam|last8=Vaswani|first8=Ashish|date=2017-06-12|title=Attention Is All You Need|eprint=1706.03762|class=cs.CL}}</ref>的实现几乎“完全一致”。<br />
<br />
BERT在两个任务上进行预训练： 语言模型（15%的token被掩盖，BERT需要从上下文中进行推断）和下一句预测（BERT需要预测给定的第二个句子是否是第一句的下一句）。训练完成后，BERT学习到单词的上下文嵌入。代价昂贵的预训练完成后，BERT可以使用较少的资源和较小的数据集在下游任务上进行微调，以改进在这些任务上的性能。<ref name=":0"/><ref>{{cite web |last1=Horev |first1=Rani |title=BERT Explained: State of the art language model for NLP |url=https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270 |website=Towards Data Science |access-date=27 September 2021 |date=2018 |archive-date=2022-10-17 |archive-url=https://web.archive.org/web/20221017103715/https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270 |dead-url=no }}</ref><br />
<br />
== 性能及分析 ==<br />
BERT在以下[[自然语言理解]]任务上的性能表现得最为卓越：<ref name=":0" /><br />
* GLUE（General Language Understanding Evaluation，通用语言理解评估）任务集（包括9个任务）。<br />
* SQuAD（Stanford Question Answering Dataset，斯坦福问答数据集）v1.1和v2.0。<br />
* SWAG（Situations With Adversarial Generation，对抗生成的情境）。<br />
<br />
有關BERT在上述自然语言理解任务中為何可以達到先进水平，目前還未找到明確的原因<ref name=":1">{{Cite book|last1=Kovaleva|first1=Olga|last2=Romanov|first2=Alexey|last3=Rogers|first3=Anna|last4=Rumshisky|first4=Anna|date=November 2019|chapter=Revealing the Dark Secrets of BERT|chapter-url=https://www.aclweb.org/anthology/D19-1445|language=en-us|pages=4364–4373|doi=10.18653/v1/D19-1445|title=Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)|access-date=2020-10-19|archive-date=2020-10-20|archive-url=https://web.archive.org/web/20201020075649/https://www.aclweb.org/anthology/D19-1445/|dead-url=no}}</ref><ref name=":2">{{Cite journal|last1=Clark|first1=Kevin|last2=Khandelwal|first2=Urvashi|last3=Levy|first3=Omer|last4=Manning|first4=Christopher D.|date=2019|title=What Does BERT Look at? An Analysis of BERT's Attention|journal=Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP|pages=276–286|location=Stroudsburg, PA, USA|publisher=Association for Computational Linguistics}}</ref>。目前BERT的可解释性研究主要集中在研究精心选择的输入序列对BERT的输出的影响关系，<ref>{{Cite journal|last1=Khandelwal|first1=Urvashi|last2=He|first2=He|last3=Qi|first3=Peng|last4=Jurafsky|first4=Dan|date=2018|title=Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context|journal=Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)|pages=284–294|location=Stroudsburg, PA, USA|publisher=Association for Computational Linguistics|doi=10.18653/v1/p18-1027|bibcode=2018arXiv180504623K|arxiv=1805.04623}}</ref><ref>{{Cite journal|last1=Gulordava|first1=Kristina|last2=Bojanowski|first2=Piotr|last3=Grave|first3=Edouard|last4=Linzen|first4=Tal|last5=Baroni|first5=Marco|date=2018|title=Colorless Green Recurrent Networks Dream Hierarchically|journal=Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)|pages=1195–1205|location=Stroudsburg, PA, USA|publisher=Association for Computational Linguistics|doi=10.18653/v1/n18-1108|bibcode=2018arXiv180311138G|arxiv=1803.11138}}</ref>通过探测分类器分析内部[[向量空間模型|向量表示]]，<ref>{{Cite journal|last1=Giulianelli|first1=Mario|last2=Harding|first2=Jack|last3=Mohnert|first3=Florian|last4=Hupkes|first4=Dieuwke|last5=Zuidema|first5=Willem|date=2018|title=Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information|journal=Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP|pages=240–248|location=Stroudsburg, PA, USA|publisher=Association for Computational Linguistics|doi=10.18653/v1/w18-5426|bibcode=2018arXiv180808079G|arxiv=1808.08079}}</ref><ref>{{Cite journal|last1=Zhang|first1=Kelly|last2=Bowman|first2=Samuel|date=2018|title=Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis|journal=Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP|pages=359–361|location=Stroudsburg, PA, USA|publisher=Association for Computational Linguistics|doi=10.18653/v1/w18-5448}}</ref>以及注意力权重表示的关系。<ref name=":1" /><ref name=":2" /><br />
<br />
== 历史 ==<br />
BERT起源于预训练的上下文表示学习，包括[[半监督序列学习]]（Semi-supervised Sequence Learning）<ref>{{cite arxiv |last1=Dai |first1=Andrew |last2=Le | first2=Quoc |title=Semi-supervised Sequence Learning |date=2015-11-04 |eprint=1511.01432|class=cs.LG }}</ref>，[[生成预训练]]（Generative Pre-Training），{{le|ELMo|ELMo}}<ref>{{cite arxiv |last1=Peters |first1=Matthew |last2=Neumann | first2=Mark |last3=Iyyer | first3=Mohit |last4=Gardner | first4=Matt | last5=Clark | first5=Christopher | last6=Lee | first6=Kenton | last7=Luke | first7= Zettlemoyer |title=Deep contextualized word representations |date=2018-02-15 |eprint=1802.05365v2|class=cs.CL }}</ref>和[[ULMFit]]<ref>{{cite arxiv |last1=Howard |first1=Jeremy |last2=Ruder | first2=Sebastian |title=Universal Language Model Fine-tuning for Text Classification |date=2018-01-18 |eprint=1801.06146v5|class=cs.CL }}</ref>。与之前的模型不同，BERT是一种深度双向的、无监督的语言表示，且仅使用纯文本语料库进行预训练的模型。上下文无关模型（如[[word2vec]]或{{le|GloVe|GloVe}}）为词汇表中的每个单词生成一个词向量表示，因此容易出现单词的歧义问题。BERT考虑到单词出现时的上下文。例如，词“水分”的word2vec词向量在“植物需要吸收水分”和“财务报表裡有水分”是相同的，但BERT根据上下文的不同提供不同的词向量，词向量与句子表达的句意有关。<br />
<br />
2019年10月25日，[[Google搜索]]宣布他们已经开始在美国国内的英语搜索查询中应用BERT模型。<ref>{{cite web |last1=Nayak |first1=Pandu |title=Understanding searches better than ever before |url=https://www.blog.google/products/search/search-language-understanding-bert/ |website=Google Blog |date=2019-10-25 |accessdate=2019-12-10 |archive-date=2019-12-05 |archive-url=https://web.archive.org/web/20191205195841/https://www.blog.google/products/search/search-language-understanding-bert/ |dead-url=no }}</ref>2019年12月9日，据报道，Google搜索已经在70多种语言的搜索采用了BERT。<ref>{{cite web |last1=Montti |first1=Roger |title=Google's BERT Rolls Out Worldwide |url=https://www.searchenginejournal.com/google-bert-rolls-out-worldwide/339359/ |website=Search Engine Journal |date=2019-12-10 |publisher=Search Engine Journal |accessdate=2019-12-10 |archive-date=2020-11-29 |archive-url=https://web.archive.org/web/20201129083635/https://www.searchenginejournal.com/google-bert-rolls-out-worldwide/339359/ |dead-url=no }}</ref>2020年10月，几乎每一个基于英语的查询都由BERT处理。<ref>{{Cite web|date=2020-10-15|title=Google: BERT now used on almost every English query|url=https://searchengineland.com/google-bert-used-on-almost-every-english-query-342193|access-date=2020-11-24|website=Search Engine Land|archive-date=2022-05-06|archive-url=https://web.archive.org/web/20220506220519/https://searchengineland.com/google-bert-used-on-almost-every-english-query-342193}}</ref><br />
<br />
==获奖情况==<br />
在2019年[[计算语言学协会]]北美分会（{{le|计算语言学协会北美分会|North American Chapter of the Association for Computational Linguistics|NAACL}}）年会上，BERT获得了最佳长篇论文奖。<ref>{{Cite web|url=https://naacl2019.org/blog/best-papers/|title=Best Paper Awards|last=|first=|date=2019|website=NAACL|dead-url=no|archive-url=https://web.archive.org/web/20201019222406/https://naacl2019.org/blog/best-papers/|archive-date=2020-10-19|access-date=2020-03-28}}</ref><br />
<br />
==参见==<br />
{{div col|colwidth=18em}}<br />
* [[Transformer模型]]<br />
* [[Word2vec]]<br />
* [[自编码器]]<br />
* {{le|文献-检索词矩阵|Document-term matrix}}<br />
* [[特征提取]]<br />
* [[特征学习]]<br />
* {{le|神经网络语言模型|Neural network language model}}<br />
* [[向量空间模型]]<br />
* {{le|概念向量|Thought vector}}<br />
* {{le|fastText}}<br />
* {{le|GloVe}}<br />
* [[TensorFlow]]<br />
{{div col end}}<br />
<br />
==参考文献==<br />
{{reflist|2}}<br />
<br />
==外部链接==<br />
* [https://github.com/google-research/bert 官方GitHub仓库] {{Wayback|url=https://github.com/google-research/bert |date=20210113211317 }}<br />
<br />
{{自然语言处理}}<br />
{{Differentiable computing}}<br />
{{Google AI}}<br />
<br />
[[Category:自然语言处理]]<br />
[[Category:计算语言学]]<br />
[[Category:语音识别]]<br />
[[Category:人工智能]]</div>imported>Kiyoteru Awaji