imported>Ohtashinichiro：/* 外部連結 */

2024-09-05T00:39:25Z

外部連結

新页面

{{NoteTA|G1=IT|G2=FL}}
{{Infobox software
| name = Apache Hadoop
| logo = Hadoop logo.svg
| logo_alt = Hadoop Logo
| screenshot =
| caption =
| author = [[Doug Cutting]], [[Mike Cafarella]]
| developer = [[Apache软件基金会]]
| released = {{Start date and age|2006|04|01}}<ref>{{cite web |url=https://archive.apache.org/dist/hadoop/common/ |title=Hadoop Releases |website=apache.org |publisher=Apache Software Foundation |accessdate=2019-04-28 |archive-date=2019-04-28 |archive-url=https://web.archive.org/web/20190428190426/https://archive.apache.org/dist/hadoop/common/ |dead-url=no }}</ref>
| operating system = [[跨平台]]
| programming language = [[Java]]
| genre = [[大數據]]、[[分佈式系統]]
| license = [[Apache許可證]] 2.0
| website = {{URL|https://hadoop.apache.org/}}
}}

'''Apache Hadoop'''是一款支持數據密集型[[分布式计算|分佈式]]應用程序并以[[Apache许可证|Apache 2.0許可協議]]發佈的[[開源]][[軟體框架]]，有助于使用许多计算机组成的网络来解决数据、计算密集型的问题。基于[[MapReduce]]计算模型，它为[[大数据]]的[[集群文件系统|分布式存储]]与处理提供了一个[[软件框架]]。所有的Hadoop模块都有一个基本假设，即硬件故障是常见情况，应该由框架自动处理<ref name="homepage">{{Cite web|title= Welcome to Apache Hadoop!|url= http://hadoop.apache.org/|website= hadoop.apache.org|access-date= 2016-08-25|archive-date= 2017-09-23|archive-url= https://web.archive.org/web/20170923024639/http://hadoop.apache.org/|dead-url= no}}</ref>。

Apache Hadoop的核心模块分为存储和计算模块，前者被称为Hadoop分布式文件系统（HDFS），后者即MapReduce计算模型。Hadoop框架先将文件分成数据块并分布式地存储在集群的计算节点中，接着将负责计算任务的代码传送给各节点，让其能够并行地处理数据。这种方法有效利用了[[数据局部性]]，令各节点分别处理其能够访问的数据。与传统的[[超级计算机]]架构相比，这使得数据集的处理速度更快、效率更高<ref>{{cite web |url=http://www.datascienceassn.org/content/data-locality-hpc-vs-hadoop-vs-spark |title=Data Locality: HPC vs. Hadoop vs. Spark |last1=Malak |first1=Michael |date=2014-09-19 |website=datascienceassn.org |publisher=Data Science Association |access-date=2014-10-30 |archive-date=2017-09-10 |archive-url=https://web.archive.org/web/20170910220353/http://www.datascienceassn.org/content/data-locality-hpc-vs-hadoop-vs-spark |dead-url=no }}</ref><ref>{{cite book|chapter=Characterization and Optimization of Memory-Resident MapReduce on HPC Systems|publisher=IEEE|date=October 2014|doi=10.1109/IPDPS.2014.87|title=2014 IEEE 28th International Parallel and Distributed Processing Symposium|pages=799–808|last1=Wang|first1=Yandong|last2=Goldstone|first2=Robin|last3=Yu|first3=Weikuan|last4=Wang|first4=Teng|isbn=978-1-4799-3800-1|s2cid=11157612}}</ref>。

Apache Hadoop框架由以下基本模块构成：
* ''Hadoop Common'' – 包含了其他Hadoop 模块所需的库和实用程序；
* ''Hadoop Distributed File System (HDFS)'' – 一种将数据存储在集群中多个节点中的分布式文件系统，能够提供很高的带宽；
* ''Hadoop YARN'' – （于2012年引入）一个负责管理集群中计算资源，并实现用户程序调度的平台<ref>{{cite web |url=http://hadoop.apache.org/docs/r2.5.1/api/org/apache/hadoop/yarn/api/records/Resource.html#newInstance(int,%20int) |title=Resource (Apache Hadoop Main 2.5.1 API)  |date=2014-09-12 |website=apache.org |publisher=Apache Software Foundation |access-date=2014-09-30 |archive-url=https://web.archive.org/web/20141006090717/http://hadoop.apache.org/docs/r2.5.1/api/org/apache/hadoop/yarn/api/records/Resource.html#newInstance(int,%20int) |archive-date=2014-10-06 |url-status=dead }}</ref><ref>{{cite web |url=http://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/ |title=Apache Hadoop YARN – Concepts and Applications |last1=Murthy |first1=Arun |date=2012-08-15 |website=hortonworks.com |publisher=Hortonworks |access-date=2014-09-30 |archive-date=2017-09-11 |archive-url=https://web.archive.org/web/20170911024811/https://hortonworks.com/blog/apache-hadoop-yarn-concepts-and-applications/ |dead-url=no }}</ref>；
* ''Hadoop MapReduce'' – 用于大规模数据处理的MapReduce计算模型实现；
* ''Hadoop Ozone'' – （于2020年引入） Hadoop的对象存储。

''Hadoop'' 一词通常代指其基本模块和子模块以及生态系统<ref>{{cite web |url=https://finance.yahoo.com/news/continuuity-raises-10-million-series-120500471.html |title=Continuuity Raises $10 Million Series A Round to Ignite Big Data Application Development Within the Hadoop Ecosystem  |date=2012-11-14 |website=finance.yahoo.com |publisher=[[Marketwired]] |access-date=2014-10-30 |archive-date=2017-09-10 |archive-url=https://web.archive.org/web/20170910221558/https://finance.yahoo.com/news/continuuity-raises-10-million-series-120500471.html |dead-url=no }}</ref>，或可以安装在 Hadoop 之上的软件包的集合，例如[[Pig|Apache Pig]]、[[Apache Hive]]、[[Apache HBase]]、[[Apache Phoenix]]、[[Apache Spark]]、[[Apache ZooKeeper]]、[[Cloudera Impala]]、[[Apache Flume]]、[[Apache Sqoop]]、[[Apache Oozie]]和[[Apache Storm]]<ref>{{cite web |url=http://hadoop.apache.org/ |title=Hadoop-related projects at |publisher=Hadoop.apache.org |access-date=2013-10-17 |archive-date=2017-09-23 |archive-url=https://web.archive.org/web/20170923024639/http://hadoop.apache.org/ |dead-url=no }}</ref>。

Apache Hadoop的MapReduce和HDFS模块的灵感来源于[[Google]]的[[MapReduce]]和[[Google File System]]论文<ref>{{cite book  |title=Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data |url=https://books.google.com/books?id=axruBQAAQBAJ&pg=PA300|publisher=John Wiley & Sons |page=300 |date=2014-12-19 |isbn=9781118876220 |access-date=2015-01-29 }}</ref>。

Hadoop 框架本身主要是用[[Java]]编程语言编写的，也包括了一些[[C语言]]编写的本机代码和[[Shell脚本]]编写的[[命令行界面|命令行]]实用程序。尽管MapReduce Java代码很常见，但任何编程语言都可以与Hadoop Streaming一起使用来实现用户程序的map和reduce部分<ref>{{cite web |url=http://www.mail-archive.com/nlpatumd@yahoogroups.com/msg00570.html |title=nlpatumd; Adventures with Hadoop and Perl |publisher=Mail-archive.com |date=2010-05-02 |access-date=2013-04-05 |archive-date=2017-08-14 |archive-url=https://web.archive.org/web/20170814193116/http://www.mail-archive.com/nlpatumd@yahoogroups.com/msg00570.html |dead-url=no }}</ref>。Hadoop 生态系统中的其他项目实现了更为丰富的用户界面。

== 主要子项目 ==
[[File:Cubieboard HADOOP cluster.JPG|thumb|Hadoop小[[计算机集群]]用Cubieboard电脑。]]
* Hadoop Common：在0.20及以前的版本中，包含HDFS、[[MapReduce]]和其他项目公共内容，从0.21开始HDFS和MapReduce被分离为独立的子项目，其余内容为Hadoop Common
* HDFS：Hadoop分佈式文件系統（Distributed File System）－HDFS（Hadoop Distributed File System）
* MapReduce：并行计算框架，0.20前使用org.apache.hadoop.mapred旧接口，0.20版本开始引入org.apache.hadoop.mapreduce的新API

== 相關项目 ==
* [[Apache HBase]]：分布式[[NoSQL]]列[[数据库]]，类似[[谷歌]]公司[[BigTable]]。
* [[Apache Hive]]：构建于hadoop之上的[[数据仓库]]，通过一种类[[SQL]]语言HiveQL为用户提供数据的归纳、查询和分析等功能。Hive最初由[[Facebook]]贡献。
* [[Apache Mahout]]：[[机器学习]]算法软件包。
* [[Apache Sqoop]]：[[结构化数据]]（如[[关系数据库]]）与Apache Hadoop之间的数据转换工具。
* [[Apache ZooKeeper]]：分布式锁设施，提供类似[[Google]] [[Chubby]]的功能，由[[Facebook]]贡献。
* [[Apache Avro]]：新的数据[[序列化]]格式与传输工具，将逐步取代Hadoop原有的IPC机制。

== 知名用戶 ==
=== Hadoop在Yahoo!的應用 ===
2008年2月19日，[[雅虎]]使用10,000個[[微處理器]]核心的[[Linux]][[计算机集群]]運行一個Hadoop應用程式。<ref>{{Cite web |url=http://developer.yahoo.com/blogs/hadoop/2008/02/yahoo-worlds-largest-production-hadoop.html |title=Yahoo! Launches World's Largest Hadoop Production Application (Hadoop and Distributed Computing at Yahoo!) |accessdate=2008-09-04 |archive-date=2008-05-14 |archive-url=https://web.archive.org/web/20080514135459/http://developer.yahoo.com/blogs/hadoop/2008/02/yahoo-worlds-largest-production-hadoop.html |dead-url=yes }}</ref>

=== 其他用戶 ===
其他知名用戶包括<ref>{{Cite web |url=http://wiki.apache.org/hadoop/PoweredBy |title=PoweredBy |access-date=2008-09-07 |archive-url=https://web.archive.org/web/20121129102729/http://wiki.apache.org/hadoop/PoweredBy |archive-date=2012-11-29 |dead-url=yes }}</ref>：
<div style="-moz-column-count:3; column-count:3;">
* A9.com
* [[Facebook]]
* Fox Interactive Media
* [[华为]]
* [[IBM]]
* ImageShack
* [[資訊科學研究院]]
* [[Joost]]
* [[Last.fm]]
* Powerset
* [[紐約時報]]
* Rackspace
* Veoh
* [[中華電信]]
* [[中国移动]]
</div>

== Hadoop與Sun Grid Engine ==
[[昇陽]]電腦的[[Sun Grid Engine]]可以用来调度Hadoop Job。<ref>{{cite web|url=http://blogs.sun.com/ravee/entry/creating_hadoop_pe_under_sge|title=Creating Hadoop pe under SGE|date=2008-01-16|publisher=[[Sun Microsystems]]|deadurl=yes|archiveurl=https://web.archive.org/web/20080912170855/http://blogs.sun.com/ravee/entry/creating_hadoop_pe_under_sge|archivedate=2008-09-12|accessdate=2008-09-04}}</ref><ref>{{cite web|url=http://wikis.sun.com/download/attachments/170755116/hpcworkshop_hdfs.pdf|title=HDFS-Aware Scheduling With Grid Engine|date=2009-09-10|publisher=[[Sun Microsystems]]}}{{dead link|date=2017年11月 |bot=InternetArchiveBot |fix-attempted=yes }}</ref>

== Hadoop與Condor ==
[[威斯康辛大學麥迪遜分校]]的Condor[[計算機集群]]軟件也可以用作Hadoop Job的排程。<ref>{{cite web|url=http://www.cs.wisc.edu/condor/CondorWeek2010/condor-presentations/thain-condor-hadoop.pdf|title=Condor integrated with Hadoop's Map Reduce|date=2010-04-15|publisher=[[威斯康辛大學麥迪遜分校]]|accessdate=2011-03-15|archive-date=2011-04-01|archive-url=https://web.archive.org/web/20110401055838/http://www.cs.wisc.edu/condor/CondorWeek2010/condor-presentations/thain-condor-hadoop.pdf|dead-url=no}}</ref>

== 參見 ==
* [[大数据]]
* [[雲端運算]]
* [[高性能計算集群]]
* [[OpenStack]]－以[[Apache許可證]]授權的雲端運算軟件。
* [[Apache Spark]]

== 参考文献 ==
{{Reflist}}

== 外部連結 ==
* [http://hadoop.apache.org/ Hadoop官方網站]{{Wayback|url=http://hadoop.apache.org/ |date=20170923024639 }}

{{-}}
{{Apache}}
{{Filesystem}}
{{Authority control}}

[[Category:Apache软件基金会项目|Hadoop]]
[[Category:大數據]]
[[Category:分散式檔案系統]]
[[Category:雲端運算]]
[[Category:用Java編程的自由軟體]]
[[Category:自由系統軟體]]
[[Category:Hadoop| ]]
[[Category:使用Apache许可证的软件]]

Apache Hadoop - 版本历史

imported>Ohtashinichiro：​/* 外部連結 */

imported>Ohtashinichiro：/* 外部連結 */