- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
除了“如何使用 MapReduce 计算长文本中的单词数”任务之外,我想不出任何好的例子。我发现这并不是让其他人了解这个工具有多强大的最佳示例。
我不是在寻找代码片段,实际上只是“文本”示例。
最佳答案
MapReduce是一个为高效处理海量数据而开发的框架。例如,如果数据集中有 100 万条记录,并且它存储在关系表示中 - 派生值并对这些记录执行任何类型的转换都非常昂贵。
例如,在 SQL 中,给定出生日期,要找出一百万条记录中有多少人年龄 > 30 岁,这需要一段时间,而且当查询复杂性增加时,这只会按数量级增加。MapReduce提供了基于集群的实现,其中数据以分布式方式处理
这里有一篇维基百科文章解释了什么 map-reduce
is all about
另一个很好的例子是通过 MapReduce 寻找 friend 可以是理解这个概念的有力例子,并且一个很好用的用例。
个人发现this link对于理解这个概念非常有用
复制博客中提供的说明(以防链接失效)
Finding Friends
MapReduce is a framework originally developed at Google that allows for easy large scale distributed computing across a number of domains. Apache Hadoop is an open source implementation.
I'll gloss over the details, but it comes down to defining two functions: a map function and a reduce function. The map function takes a value and outputs key:value pairs. For instance, if we define a map function that takes a string and outputs the length of the word as the key and the word itself as the value then map(steve) would return 5:steve and map(savannah) would return 8:savannah. You may have noticed that the map function is stateless and only requires the input value to compute it's output value. This allows us to run the map function against values in parallel and provides a huge advantage. Before we get to the reduce function, the mapreduce framework groups all of the values together by key, so if the map functions output the following key:value pairs:
3 : the
3 : and
3 : you
4 : then
4 : what
4 : when
5 : steve
5 : where
8 : savannah
8 : researchThey get grouped as:
3 : [the, and, you]
4 : [then, what, when]
5 : [steve, where]
8 : [savannah, research]Each of these lines would then be passed as an argument to the reduce function, which accepts a key and a list of values. In this instance, we might be trying to figure out how many words of certain lengths exist, so our reduce function will just count the number of items in the list and output the key with the size of the list, like:
3 : 3
4 : 3
5 : 2
8 : 2The reductions can also be done in parallel, again providing a huge advantage. We can then look at these final results and see that there were only two words of length 5 in our corpus, etc...
The most common example of mapreduce is for counting the number of times words occur in a corpus. Suppose you had a copy of the internet (I've been fortunate enough to have worked in such a situation), and you wanted a list of every word on the internet as well as how many times it occurred.
The way you would approach this would be to tokenize the documents you have (break it into words), and pass each word to a mapper. The mapper would then spit the word back out along with a value of
1
. The grouping phase will take all the keys (in this case words), and make a list of 1's. The reduce phase then takes a key (the word) and a list (a list of 1's for every time the key appeared on the internet), and sums the list. The reducer then outputs the word, along with it's count. When all is said and done you'll have a list of every word on the internet, along with how many times it appeared.Easy, right? If you've ever read about mapreduce, the above scenario isn't anything new... it's the "Hello, World" of mapreduce. So here is a real world use case (Facebook may or may not actually do the following, it's just an example):
Facebook has a list of friends (note that friends are a bi-directional thing on Facebook. If I'm your friend, you're mine). They also have lots of disk space and they serve hundreds of millions of requests everyday. They've decided to pre-compute calculations when they can to reduce the processing time of requests. One common processing request is the "You and Joe have 230 friends in common" feature. When you visit someone's profile, you see a list of friends that you have in common. This list doesn't change frequently so it'd be wasteful to recalculate it every time you visited the profile (sure you could use a decent caching strategy, but then I wouldn't be able to continue writing about mapreduce for this problem). We're going to use mapreduce so that we can calculate everyone's common friends once a day and store those results. Later on it's just a quick lookup. We've got lots of disk, it's cheap.
Assume the friends are stored as Person->[List of Friends], our friends list is then:
A -> B C D
B -> A C D E
C -> A B D E
D -> A B C E
E -> B C DEach line will be an argument to a mapper. For every friend in the list of friends, the mapper will output a key-value pair. The key will be a friend along with the person. The value will be the list of friends. The key will be sorted so that the friends are in order, causing all pairs of friends to go to the same reducer. This is hard to explain with text, so let's just do it and see if you can see the pattern. After all the mappers are done running, you'll have a list like this:
For map(A -> B C D) :
(A B) -> B C D
(A C) -> B C D
(A D) -> B C D
For map(B -> A C D E) : (Note that A comes before B in the key)
(A B) -> A C D E
(B C) -> A C D E
(B D) -> A C D E
(B E) -> A C D E
For map(C -> A B D E) :
(A C) -> A B D E
(B C) -> A B D E
(C D) -> A B D E
(C E) -> A B D E
For map(D -> A B C E) :
(A D) -> A B C E
(B D) -> A B C E
(C D) -> A B C E
(D E) -> A B C E
And finally for map(E -> B C D):
(B E) -> B C D
(C E) -> B C D
(D E) -> B C D
Before we send these key-value pairs to the reducers, we group them by their keys and get:
(A B) -> (A C D E) (B C D)
(A C) -> (A B D E) (B C D)
(A D) -> (A B C E) (B C D)
(B C) -> (A B D E) (A C D E)
(B D) -> (A B C E) (A C D E)
(B E) -> (A C D E) (B C D)
(C D) -> (A B C E) (A B D E)
(C E) -> (A B D E) (B C D)
(D E) -> (A B C E) (B C D)Each line will be passed as an argument to a reducer. The reduce function will simply intersect the lists of values and output the same key with the result of the intersection. For example, reduce((A B) -> (A C D E) (B C D)) will output (A B) : (C D) and means that friends A and B have C and D as common friends.
The result after reduction is:
(A B) -> (C D)
(A C) -> (B D)
(A D) -> (B C)
(B C) -> (A D E)
(B D) -> (A C E)
(B E) -> (C D)
(C D) -> (A B E)
(C E) -> (B D)
(D E) -> (B C)Now when D visits B's profile, we can quickly look up
(B D)
and see that they have three friends in common,(A C E)
.
关于mapreduce - 良好的 MapReduce 示例,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/12375761/
我正在处理一个处理大量数据的项目,所以我最近发现了 MapReduce,在我进一步深入研究之前,我想确保我的期望是正确的。 与数据的交互将通过 Web 界面进行,因此响应时间在这里至关重要,我认为 1
我正在阅读有关 Hadoop 以及它的容错性的文章。我阅读了 HDFS 并阅读了如何处理主节点和从节点的故障。但是,我找不到任何提及 mapreduce 如何执行容错的文档。特别是,当包含 Job T
我正在尝试在我的 Ubuntu 桌面上使用最新的 Hadoop 版本 2.6.0、Java SDK 1.70 来模拟 Hadoop 环境。我用必要的环境参数配置了 hadoop,它的所有进程都已启动并
就目前情况而言,这个问题不太适合我们的问答形式。我们希望答案得到事实、引用资料或专业知识的支持,但这个问题可能会引发辩论、争论、民意调查或扩展讨论。如果您觉得这个问题可以改进并可能重新开放,visit
我只是想针对我们正在做的一些数据分析工作来评估 HBase。 HBase 将包含我们的事件数据。键为 eventId + 时间。我们想要对日期范围内的几种事件类型 (4-5) 进行分析。事件类型总数约
是否有一种快速算法可以在 MapReduce 框架上运行以从巨大的整数集中查找中位数? 最佳答案 我会这样做。这是顺序快速选择的一种并行版本。 (某些映射/归约工具可能不会让您轻松完成任务...) 从
我正在尝试对大型分布式数据集执行一些数值计算。该算法非常适合 MapReduce 模型,具有以下附加属性:与输入数据相比,映射步骤的输出尺寸较小。数据可以被视为只读,并且静态分布在节点上(故障转移时的
假设我在 RavenDb 中有给定的文档结构 public class Car { public string Manufacturer {get;set;} public int B
我刚刚开始使用 mongo 和 map/reduce,在使用 pymongo 时我遇到了以下错误,而在直接使用 mongo 命令行时我没有得到(我意识到有一个类似的问题这个,但我的似乎更基本)。 我直
*基本上我正在尝试按过去一小时内的得分对对象进行排序。 我正在尝试为我的数据库中的对象生成每小时投票总和。投票嵌入到每个对象中。对象架构如下所示: { _id: ObjectId sc
我们怎样才能使我们的 MapReduce 查询更快? 我们使用五节点 Riak 数据库集群构建了一个应用程序。 我们的数据模型由三个部分组成:比赛、联赛和球队。 比赛包含联赛和球队的链接: 型号 va
关闭。这个问题不符合Stack Overflow guidelines .它目前不接受答案。 我们不允许提问寻求书籍、工具、软件库等的推荐。您可以编辑问题,以便用事实和引用来回答。 关闭 6 年前。
有没有什么方法可以在运行时获取应用程序 ID - 例如 - 带有 yarn 的 wordcount 示例命令? 我希望使用 yarn 从另一个进程启 Action 业命令,并通过 YARN REST
如何在Hadoop Map-reduce程序中使用机器学习算法?我想使用分类算法、决策树、聚类算法。除了 Mahout 之外,请提出一些想法。 最佳答案 您可以编写自己的MapReduce程序,并在m
虽然 MapReduce 可能不是实现图像处理中使用的算法的最佳方式,但出于好奇,如果我作为初学者尝试使用它们,这将是最简单的实现方式。 最佳答案 Hadoop 非常适合处理大量 IO。因此,例如,您
我只是想验证我对这些参数及其关系的理解,如果我错了请通知我。 mapreduce.reduce.shuffle.input.buffer.percent 告诉分配给 reducer 的整个洗牌阶段的内
HBase 需要 mapreduce/yarn,还是只需要 hdfs? 对于 HBase 的基本用法,例如创建表、插入数据、扫描/获取数据,我看不出有任何理由使用 mapreduce/yarn。 请帮
我问了一些关于提高 Hive 查询性能的问题。一些答案与映射器和化简器的数量有关。我尝试了多个映射器和化简器,但在执行过程中没有发现任何差异。不知道为什么,可能是我没有以正确的方式去做,或者我错过了别
我是 mapreduce 和 hadoop 的新手。我阅读了 mapreduce 的示例和设计模式... 好的,我们可以进入正题了。我们正在开发一种软件,可以监控系统并定期捕获它们的 CPU 使用
我正在使用 Microsoft MapReduce SDK 启动仅 Mapper 作业。 调用 hadoop.MapReduceJob.ExecuteJob 立即抛出“响应状态代码不表示成功:404(
我是一名优秀的程序员,十分优秀!