org.apache.hadoop.tools.rumen.ZombieJob类的使用及代码示例-6ren

org.apache.hadoop.tools.rumen.ZombieJob类的使用及代码示例

转载作者：知者更新时间：2024-03-13 12:54:01

本文整理了Java中org.apache.hadoop.tools.rumen.ZombieJob类的一些代码示例，展示了ZombieJob类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。ZombieJob类的具体详情如下：
包路径：org.apache.hadoop.tools.rumen.ZombieJob
类名称：ZombieJob

ZombieJob介绍

[英]ZombieJob is a layer above LoggedJob raw JSON objects. Each ZombieJob object represents a job in job history. For everything that exists in job history, contents are returned unchanged faithfully. To get input splits of a non-exist task, a non-exist task attempt, or an ill-formed task attempt, proper objects are made up from statistical sketches.
[中]ZombieJob是LoggedJob原始JSON对象之上的一层。每个僵尸作业对象代表作业历史中的一个作业。对于工作历史记录中存在的所有内容，都会原封不动地返回。为了获得不存在的任务、不存在的任务尝试或格式不正确的任务尝试的输入拆分，从统计草图中生成适当的对象。

代码示例

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

private LoggedTask getLoggedTask(TaskType taskType, int taskNumber) {
 buildMaps();
 return loggedTaskMap.get(getMaskedTaskID(taskType, taskNumber));
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

private long makeUpReduceRuntime(State state) {
 long reduceTime = 0;
 for (int i = 0; i < 5; i++) {
  reduceTime = doMakeUpReduceRuntime(state);
  if (reduceTime >= 0) {
   return reduceTime;
  }
 }
 return 0;
}

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

LoggedTask loggedTask = getLoggedTask(taskType, taskNumber);
if (loggedTask == null) {
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
LoggedTaskAttempt loggedAttempt = getLoggedTaskAttempt(taskType,
  taskNumber, taskAttemptNumber);
if (loggedAttempt == null) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
} else {
  TaskInfo taskInfo = getTaskInfo(loggedTask);
  return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
    taskNumber, locality);
 } else {
  return getTaskAttemptInfo(loggedTask, loggedAttempt);

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

@Override
public TaskInfo getTaskInfo(TaskType taskType, int taskNumber) {
 return getTaskInfo(getLoggedTask(taskType, taskNumber));
}

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

int taskAttemptNumber, int locality) {
TaskType taskType = TaskType.MAP;
LoggedTask loggedTask = getLoggedTask(taskType, taskNumber);
if (loggedTask == null) {
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
LoggedTaskAttempt loggedAttempt = getLoggedTaskAttempt(taskType,
  taskNumber, taskAttemptNumber);
if (loggedAttempt == null) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
} else {
  TaskInfo taskInfo = getTaskInfo(loggedTask);
  return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
    taskNumber, locality);
 } else if (loggedAttempt.getResult() == Values.FAILED) {
  return getTaskAttemptInfo(loggedTask, loggedAttempt);
 } else if (loggedAttempt.getResult() == Values.SUCCESS) {
  int loggedLocality = getLocality(loggedTask, loggedAttempt);
  if (locality == loggedLocality) {
   return getTaskAttemptInfo(loggedTask, loggedAttempt);
  } else {

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

private TaskAttemptInfo getTaskAttemptInfo(LoggedTask loggedTask,
  LoggedTaskAttempt loggedAttempt) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 State state = convertState(loggedAttempt.getResult());
 if (loggedTask.getTaskType() == Values.MAP) {
  long taskTime;
  if (loggedAttempt.getStartTime() == 0) {
   int locality = getLocality(loggedTask, loggedAttempt);
   taskTime = makeUpMapRuntime(state, locality);
  } else {
   taskTime = loggedAttempt.getFinishTime() - loggedAttempt.getStartTime();
  taskTime = sanitizeTaskRuntime(taskTime, loggedAttempt.getAttemptID());
  return new MapTaskAttemptInfo(state, taskInfo, taskTime, allSplitVectors);
 } else if (loggedTask.getTaskType() == Values.REDUCE) {
   long reduceTime = makeUpReduceRuntime(state);
   return new ReduceTaskAttemptInfo
    (state, taskInfo, 0, 0, reduceTime, allSplitVectors);
   long mergeTime = mergeDone - shuffleDone;
   long reduceTime = finishTime - mergeDone;
   reduceTime = sanitizeTaskRuntime(reduceTime, loggedAttempt.getAttemptID());

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

@SuppressWarnings("hiding") 
private TaskAttemptInfo scaleInfo(LoggedTask loggedTask,
  LoggedTaskAttempt loggedAttempt, int locality, int loggedLocality,
  double rackLocalOverNodeLocal, double rackRemoteOverNodeLocal) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 double[] factors = new double[] { 1.0, rackLocalOverNodeLocal,
   rackRemoteOverNodeLocal };
 double scaleFactor = factors[locality] / factors[loggedLocality];
 State state = convertState(loggedAttempt.getResult());
 if (loggedTask.getTaskType() == Values.MAP) {
  long taskTime = 0;
  if (loggedAttempt.getStartTime() == 0) {
   taskTime = makeUpMapRuntime(state, locality);
  } else {
   taskTime = loggedAttempt.getFinishTime() - loggedAttempt.getStartTime();
  }
  taskTime = sanitizeTaskRuntime(taskTime, loggedAttempt.getAttemptID());
  taskTime *= scaleFactor;
  return new MapTaskAttemptInfo
   (state, taskInfo, taskTime, loggedAttempt.allSplitVectors());
 } else {
  throw new IllegalArgumentException("taskType can only be MAP: "
    + loggedTask.getTaskType());
 }
}

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

@Override
public JobID getJobID() {
 return getLoggedJob().getJobID();
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

@Override
public ZombieJob getNextJob() throws IOException {
 LoggedJob job = reader.getNext();
 if (job == null) {
  return null;
 } else if (hasRandomSeed) {
  long subRandomSeed = RandomSeedGenerator.getSeed(
     "forZombieJob" + job.getJobID(), randomSeed);
  return new ZombieJob(job, cluster, subRandomSeed);
 } else {
  return new ZombieJob(job, cluster);
 }
}

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

private int getLocality(LoggedTask loggedTask, LoggedTaskAttempt loggedAttempt) {
 int distance = cluster.getMaximumDistance();
 String rackHostName = loggedAttempt.getHostName().getValue();
 if (rackHostName == null) {
  return distance;
 }
 MachineNode mn = getMachineNode(rackHostName);
 if (mn == null) {
  return distance;
 }
 List<LoggedLocation> locations = loggedTask.getPreferredLocations();
 if (locations != null) {
  for (LoggedLocation location : locations) {
   List<NodeName> layers = location.getLayers();
   if ((layers == null) || (layers.isEmpty())) {
    continue;
   }
   String dataNodeName = layers.get(layers.size()-1).getValue();
   MachineNode dataNode = cluster.getMachineByName(dataNodeName);
   if (dataNode != null) {
    distance = Math.min(distance, cluster.distance(mn, dataNode));
   }
  }
 }
 return distance;
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

int taskAttemptNumber, int locality) {
TaskType taskType = TaskType.MAP;
LoggedTask loggedTask = getLoggedTask(taskType, taskNumber);
if (loggedTask == null) {
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
LoggedTaskAttempt loggedAttempt = getLoggedTaskAttempt(taskType,
  taskNumber, taskAttemptNumber);
if (loggedAttempt == null) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
} else {
  TaskInfo taskInfo = getTaskInfo(loggedTask);
  return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
    taskNumber, locality);
 } else if (loggedAttempt.getResult() == Values.FAILED) {
  return getTaskAttemptInfo(loggedTask, loggedAttempt);
 } else if (loggedAttempt.getResult() == Values.SUCCESS) {
  int loggedLocality = getLocality(loggedTask, loggedAttempt);
  if (locality == loggedLocality) {
   return getTaskAttemptInfo(loggedTask, loggedAttempt);
  } else {

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

private TaskAttemptInfo getTaskAttemptInfo(LoggedTask loggedTask,
  LoggedTaskAttempt loggedAttempt) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 State state = convertState(loggedAttempt.getResult());
 if (loggedTask.getTaskType() == Values.MAP) {
  long taskTime;
  if (loggedAttempt.getStartTime() == 0) {
   int locality = getLocality(loggedTask, loggedAttempt);
   taskTime = makeUpMapRuntime(state, locality);
  } else {
   taskTime = loggedAttempt.getFinishTime() - loggedAttempt.getStartTime();
  taskTime = sanitizeTaskRuntime(taskTime, loggedAttempt.getAttemptID());
  return new MapTaskAttemptInfo(state, taskInfo, taskTime, allSplitVectors);
 } else if (loggedTask.getTaskType() == Values.REDUCE) {
   long reduceTime = makeUpReduceRuntime(state);
   return new ReduceTaskAttemptInfo
    (state, taskInfo, 0, 0, reduceTime, allSplitVectors);
   long mergeTime = mergeDone - shuffleDone;
   long reduceTime = finishTime - mergeDone;
   reduceTime = sanitizeTaskRuntime(reduceTime, loggedAttempt.getAttemptID());

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

@SuppressWarnings("hiding") 
private TaskAttemptInfo scaleInfo(LoggedTask loggedTask,
  LoggedTaskAttempt loggedAttempt, int locality, int loggedLocality,
  double rackLocalOverNodeLocal, double rackRemoteOverNodeLocal) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 double[] factors = new double[] { 1.0, rackLocalOverNodeLocal,
   rackRemoteOverNodeLocal };
 double scaleFactor = factors[locality] / factors[loggedLocality];
 State state = convertState(loggedAttempt.getResult());
 if (loggedTask.getTaskType() == Values.MAP) {
  long taskTime = 0;
  if (loggedAttempt.getStartTime() == 0) {
   taskTime = makeUpMapRuntime(state, locality);
  } else {
   taskTime = loggedAttempt.getFinishTime() - loggedAttempt.getStartTime();
  }
  taskTime = sanitizeTaskRuntime(taskTime, loggedAttempt.getAttemptID());
  taskTime *= scaleFactor;
  return new MapTaskAttemptInfo
   (state, taskInfo, taskTime, loggedAttempt.allSplitVectors());
 } else {
  throw new IllegalArgumentException("taskType can only be MAP: "
    + loggedTask.getTaskType());
 }
}

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

@Override
public TaskInfo getTaskInfo(TaskType taskType, int taskNumber) {
 return getTaskInfo(getLoggedTask(taskType, taskNumber));
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

@Override
public JobID getJobID() {
 return getLoggedJob().getJobID();
}

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

@Override
public ZombieJob getNextJob() throws IOException {
 LoggedJob job = reader.getNext();
 if (job == null) {
  return null;
 } else if (hasRandomSeed) {
  long subRandomSeed = RandomSeedGenerator.getSeed(
     "forZombieJob" + job.getJobID(), randomSeed);
  return new ZombieJob(job, cluster, subRandomSeed);
 } else {
  return new ZombieJob(job, cluster);
 }
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

private int getLocality(LoggedTask loggedTask, LoggedTaskAttempt loggedAttempt) {
 int distance = cluster.getMaximumDistance();
 String rackHostName = loggedAttempt.getHostName().getValue();
 if (rackHostName == null) {
  return distance;
 }
 MachineNode mn = getMachineNode(rackHostName);
 if (mn == null) {
  return distance;
 }
 List<LoggedLocation> locations = loggedTask.getPreferredLocations();
 if (locations != null) {
  for (LoggedLocation location : locations) {
   List<NodeName> layers = location.getLayers();
   if ((layers == null) || (layers.isEmpty())) {
    continue;
   }
   String dataNodeName = layers.get(layers.size()-1).getValue();
   MachineNode dataNode = cluster.getMachineByName(dataNodeName);
   if (dataNode != null) {
    distance = Math.min(distance, cluster.distance(mn, dataNode));
   }
  }
 }
 return distance;
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-rumen

LoggedTask loggedTask = getLoggedTask(taskType, taskNumber);
if (loggedTask == null) {
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
LoggedTaskAttempt loggedAttempt = getLoggedTaskAttempt(taskType,
  taskNumber, taskAttemptNumber);
if (loggedAttempt == null) {
 TaskInfo taskInfo = getTaskInfo(loggedTask);
 return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
   taskNumber, locality);
} else {
  TaskInfo taskInfo = getTaskInfo(loggedTask);
  return makeUpTaskAttemptInfo(taskType, taskInfo, taskAttemptNumber,
    taskNumber, locality);
 } else {
  return getTaskAttemptInfo(loggedTask, loggedAttempt);

代码示例来源：origin: ch.cern.hadoop/hadoop-rumen

private LoggedTask getLoggedTask(TaskType taskType, int taskNumber) {
 buildMaps();
 return loggedTaskMap.get(getMaskedTaskID(taskType, taskNumber));
}

代码示例来源：origin: org.apache.hadoop/hadoop-mapred-test

job = parser.getNextJob();
ZombieJob zJob = (ZombieJob) job;
LoggedJob loggedJob = zJob.getLoggedJob();
System.out.println(i + ":" + job.getNumberMaps() + "m, "
  + job.getNumberReduces() + "r");

文章推荐： de.dytanic.cloudnet.lib.zip.ZipConverter类的使用及代码示例

c - 我尝试理解 [c 代码 -> 汇编] 代码
我尝试理解[c代码 -> 汇编]代码 void node::Check( data & _data1, vector& _data2) { -> push ebp -> mov ebp,esp ->
c# - 在当前表单(代码)的上下文中从字符串动态运行 C# 代码
我需要在当前表单(代码)的上下文中运行文本文件中的代码。其中一项要求是让代码创建新控件并将其添加到当前窗体。例如，在Form1.cs中: using System.Windows.Forms; ..
c# - c++代码(malloc方法)到c#代码
我有此 C++ 代码并将其转换为 C# (.net Framework 4) 代码。有没有人给我一些关于 malloc、free 和 sprintf 方法的提示？ int monate = ee; d
C 代码，简单的 Web 服务器(代码 OK)
我的网络服务器代码有问题 #include #include #include #include #include #include #include int
html - 将特定列表元素置于斜体的 CSS 代码(不更改 html 代码)
给定以下 html 代码，将列表中的第三个元素(即“美丽”一词)以斜体显示的 CSS 代码是什么？当然，我可以给这个元素一个 id 或一个 class，但 html 代码必须保持不变。谢谢
javascript - 是否有一些库可用于 IQR 代码(不是 QR 代码)？
关闭。这个问题不符合Stack Overflow guidelines .它目前不接受答案。我们不允许提问寻求书籍、工具、软件库等的推荐。您可以编辑问题，以便用事实和引用来回答。关闭 7 年前。
macros - 在 Inno Setup [代码] 部分将宏扩展为 Pascal 代码
我试图制作一个宏来避免重复代码和注释。我试过这个: #define GrowOnPage(any Page, any Component) Component.Width := Page.Surfa
c# - 我正在尝试将我的旧 c++ 代码 "translate"转换为 c# 代码
我正在尝试将我的旧 C++ 代码“翻译”成头条新闻所暗示的 C# 代码。问题是我是 C# 中的新手，并不是所有的东西都像 C++ 中那样。在 C++ 中这些解决方案运行良好，但在 C# 中只是不能。我
r - 让 Visual Studio 代码(自动)格式化 R 代码
在 Windows 10 上工作，R 语言的格式化程序似乎没有在 Visual Studio Code 中完成它的工作。我试过R support for Visual Studio Code和 R-T
dynamic - 是否可以在 Python 脚本中生成和执行 Python 代码？ [动态 Python 代码]
我正在处理一些报告(计数)，我必须获取不同参数的计数。非常简单但乏味。一个参数的示例查询: qCountsEmployee = ( "select count(*) from %s wher
ios - 随机和偶然的网络错误(NSURLErrorDomain 代码=-1001 和 NSURLErrorDomain 代码=-1005)
最近几天我尝试从 d00m 调试网络错误。我开始用尽想法/线索，我希望其他 SO 用户拥有可能有用的宝贵经验。我希望能够提供所有相关信息，但我个人无法控制服务器环境。整个事情始于用户注意到我们应用程
javascript - visual studio 代码 intellisense 不适用于 dojo amd 代码
我有一个 app.js 文件，其中包含如下 dojo amd 模式代码: require(["dojo/dom", ..], function(dom){ dom.byId('someId').i
cuda - 'code=sm_X' 是否仅嵌入二进制(cubin)代码，或 PTX 代码，或两者？
我对“-gencode”语句中的“code=sm_X”选项有点困惑。一个例子:NVCC 编译器选项有什么作用 -gencode arch=compute_13,code=sm_13 嵌入库中？只有
javascript - 在 Javascript 下拉列表中添加 HTML 代码，Javascript 不评估 HTML 代码
我为我的表格使用 X-editable 框架。但是我有一些问题。 $(document).ready(function() { $('.access').editable({
python - 在 linux 中运行 flask/python 代码？基本 flask 代码
我一直在通过本教程学习 flask/python http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-wo
vim - G 代码 M 代码 VI 和 EMACS 的 CNC 语法
我想将 Vim 和 EMACS 用于 CNC、G 代码和 M 代码。 Vim 或 EMACS 是否有任何语法或模式来处理这种类型的代码？最佳答案一些快速搜索使我找到了 this vim 和 thi
iphone - 寻找关于将 Pre-Storyboard 代码 (XCode4) 移动到 Storyboard 代码 (XCode5) 的教程
关闭。这个问题不符合Stack Overflow guidelines .它目前不接受答案。想改进这个问题？更新问题，使其成为 on-topic对于堆栈溢出。 7年前关闭。 Improve this
vim - 如何让 Vim 理解 *.md 文件包含 Markdown 代码，而不是 Modula-2 代码？
这个问题在这里已经有了答案: Enabling markdown highlighting in Vim (5 个回答) 6年前关闭。当我在 Vim 中编辑包含 Markdown 代码的 READM
ios - 错误域=AVFoundationErrorDomain 代码=-11800 "The operation could not be completed"{错误域=NSOSStatusErrorDomain 代码=-16976 "(null)"}
我正在 Swift3 iOS 中开发视频应用程序。基本上我必须将视频 Assets 和音频与淡入淡出效果合并为一个并将其保存到 iPhone 画廊。为此，我使用以下方法: private func d
jenkins - 无法通过 Jenkins 管道作业的 jenkinsfile 中的 Groovy 代码(或 java 代码)创建文件
pipeline { agent any stages { stage('Build') { steps { e

知者

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城