gpt4 book ai didi

org.apache.hadoop.yarn.client.api.YarnClient.submitApplication()方法的使用及代码示例

转载 作者:知者 更新时间:2024-03-17 20:32:40 27 4
gpt4 key购买 nike

本文整理了Java中org.apache.hadoop.yarn.client.api.YarnClient.submitApplication()方法的一些代码示例,展示了YarnClient.submitApplication()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。YarnClient.submitApplication()方法的具体详情如下:
包路径:org.apache.hadoop.yarn.client.api.YarnClient
类名称:YarnClient
方法名:submitApplication

YarnClient.submitApplication介绍

[英]Submit a new application to YARN. It is a blocking call - it will not return ApplicationId until the submitted application is submitted successfully and accepted by the ResourceManager.

Users should provide an ApplicationId as part of the parameter ApplicationSubmissionContext when submitting a new application, otherwise it will throw the ApplicationIdNotProvidedException.

This internally calls ApplicationClientProtocol#submitApplication(SubmitApplicationRequest), and after that, it internally invokes ApplicationClientProtocol#getApplicationReport(GetApplicationReportRequest) and waits till it can make sure that the application gets properly submitted. If RM fails over or RM restart happens before ResourceManager saves the application's state, ApplicationClientProtocol#getApplicationReport(GetApplicationReportRequest) will throw the ApplicationNotFoundException. This API automatically resubmits the application with the same ApplicationSubmissionContext when it catches the ApplicationNotFoundException
[中]将新应用程序提交到YARN.这是一个阻止调用-在提交的应用程序成功提交并被ResourceManager接受之前,它不会返回ApplicationId。
提交新应用程序时,用户应在参数ApplicationSubmissionContext中提供ApplicationId,否则它将抛出ApplicationDNotProvidedException。
这会在内部调用ApplicationClientProtocol#submitApplication(SubmitApplicationRequest),然后在内部调用ApplicationClientProtocol#getApplicationReport(GetApplicationReportRequest),并等待直到能够确保正确提交应用程序。如果RM故障切换或RM重启发生在ResourceManager保存应用程序状态之前,ApplicationClientProtocol#getApplicationReport(GetApplicationReportRequest)将抛出ApplicationNotFoundException。当捕获到ApplicationNotFoundException时,此API会自动使用相同的ApplicationSubmissionContext重新提交应用程序

代码示例

代码示例来源:origin: apache/drill

public void submitAppMaster(AppSpec spec) throws YarnClientException {
 if (app == null) {
  throw new IllegalStateException("call createAppMaster( ) first");
 }
 ApplicationSubmissionContext appContext;
 try {
  appContext = spec.createAppLaunchContext(conf, app);
 } catch (IOException e) {
  throw new YarnClientException("Create app launch context failed", e);
 }
 // Submit application
 try {
  yarnClient.submitApplication(appContext);
 } catch (YarnException | IOException e) {
  throw new YarnClientException("Submit application failed", e);
 }
}

代码示例来源:origin: apache/flink

Runtime.getRuntime().addShutdownHook(deploymentFailureHook);
LOG.info("Submitting application master " + appId);
yarnClient.submitApplication(appContext);

代码示例来源:origin: alibaba/jstorm

throw new IOException(JOYConstants.INSTANCE_DEPLOY_DIR_KEY + " is not set");
jstormClientContext.yarnClient.submitApplication(appContext);

代码示例来源:origin: Qihoo360/XLearning

applicationId = yarnClient.submitApplication(applicationContext);
isRunning.set(applicationId != null);
if (isRunning.get()) {

代码示例来源:origin: apache/ignite

yarnClient.submitApplication(appContext);

代码示例来源:origin: apache/incubator-gobblin

this.yarnClient.submitApplication(appSubmissionContext);

代码示例来源:origin: uber/AthenaX

yarnClient.submitApplication(appContext);

代码示例来源:origin: apache/metron

yarnClient.submitApplication(appContext);

代码示例来源:origin: org.apache.hadoop/hadoop-mapreduce-client-jobclient

@Override
public ApplicationId
  submitApplication(ApplicationSubmissionContext appContext)
    throws YarnException, IOException {
 return client.submitApplication(appContext);
}

代码示例来源:origin: com.github.jiayuhan-it/hadoop-mapreduce-client-jobclient

@Override
public ApplicationId
  submitApplication(ApplicationSubmissionContext appContext)
    throws YarnException, IOException {
 return client.submitApplication(appContext);
}

代码示例来源:origin: org.apache.reef/reef-runtime-yarn

void submit() throws IOException, YarnException {
 LOG.log(Level.INFO, "Submitting REEF Application with UNMANAGED AM to YARN. ID: {0}", this.applicationId);
 this.yarnClient.submitApplication(this.applicationSubmissionContext);
 final Token<AMRMTokenIdentifier> token = this.yarnClient.getAMRMToken(this.applicationId);
 this.yarnProxyUser.set("reef-uam-proxy", UserGroupInformation.getCurrentUser(), token);
 this.tokenProvider.addTokens(UserCredentialSecurityTokenProvider.serializeToken(token));
}

代码示例来源:origin: org.apache.tez/tez-api

@Override
public ApplicationId submitApplication(ApplicationSubmissionContext appSubmissionContext)
  throws YarnException, IOException, TezException {
 ApplicationId appId= yarnClient.submitApplication(appSubmissionContext);
 ApplicationReport appReport = getApplicationReport(appId);
 if (appReport.getYarnApplicationState() == YarnApplicationState.FAILED){
  throw new TezException("Failed to submit application to YARN"
    + ", applicationId=" + appId
    + ", diagnostics=" + appReport.getDiagnostics());
 }
 return appId;
}

代码示例来源:origin: org.apache.twill/twill-yarn

@Override
 public ProcessController<YarnApplicationReport> submit(YarnLaunchContext context) {
  ContainerLaunchContext launchContext = context.getLaunchContext();
  YarnClient yarnClient = createYarnClient();
  try {
   addRMToken(launchContext, yarnClient, appId);
   appSubmissionContext.setAMContainerSpec(launchContext);
   appSubmissionContext.setResource(capability);
   configureAppSubmissionContext(appSubmissionContext);
   yarnClient.submitApplication(appSubmissionContext);
   return new ProcessControllerImpl(appId);
  } catch (YarnException | IOException e) {
   throw new RuntimeException("Failed to submit application " + appId, e);
  } finally {
   yarnClient.stop();
  }
 }
};

代码示例来源:origin: apache/twill

@Override
 public ProcessController<YarnApplicationReport> submit(YarnLaunchContext context) {
  ContainerLaunchContext launchContext = context.getLaunchContext();
  YarnClient yarnClient = createYarnClient();
  try {
   addRMToken(launchContext, yarnClient, appId);
   appSubmissionContext.setAMContainerSpec(launchContext);
   appSubmissionContext.setResource(capability);
   configureAppSubmissionContext(appSubmissionContext);
   yarnClient.submitApplication(appSubmissionContext);
   return new ProcessControllerImpl(appId);
  } catch (YarnException | IOException e) {
   throw new RuntimeException("Failed to submit application " + appId, e);
  } finally {
   yarnClient.stop();
  }
 }
};

代码示例来源:origin: org.apache.reef/reef-runtime-yarn

public void submit() throws IOException, YarnException {
 // SET EXEC COMMAND
 final List<String> launchCommand = new JavaLaunchCommandBuilder(launcherClazz, commandPrefixList)
   .setConfigurationFilePaths(configurationFilePaths)
   .setClassPath(this.classpath.getDriverClasspath())
   .setMemory(this.applicationSubmissionContext.getResource().getMemory())
   .setStandardOut(driverStdoutFilePath)
   .setStandardErr(driverStderrFilePath)
   .build();
 if (this.applicationSubmissionContext.getKeepContainersAcrossApplicationAttempts() &&
   this.applicationSubmissionContext.getMaxAppAttempts() == 1) {
  LOG.log(Level.WARNING, "Application will not be restarted even though preserve evaluators is set to true" +
    " since the max application submissions is 1. Proceeding to submit application...");
 }
 final ContainerLaunchContext containerLaunchContext = YarnTypes.getContainerLaunchContext(
   launchCommand, this.resources, tokenProvider.getTokens());
 this.applicationSubmissionContext.setAMContainerSpec(containerLaunchContext);
 LOG.log(Level.INFO, "Submitting REEF Application to YARN. ID: {0}", this.applicationId);
 if (LOG.isLoggable(Level.INFO)) {
  LOG.log(Level.INFO, "REEF app command: {0}", StringUtils.join(launchCommand, ' '));
 }
 this.yarnClient.submitApplication(applicationSubmissionContext);
 if (this.isUnmanaged) {
  // For Unmanaged AM mode, add a new app token to the
  // current process so it can talk to the RM as an AM.
  final Token<AMRMTokenIdentifier> token = this.yarnClient.getAMRMToken(this.applicationId);
  this.yarnProxyUser.set("reef-proxy", UserGroupInformation.getCurrentUser(), token);
  this.tokenProvider.addTokens(UserCredentialSecurityTokenProvider.serializeToken(token));
 }
}

代码示例来源:origin: ml.shifu/guagua-yarn

LOG.info("Submitting application to ASM");
  this.setAppId(this.yarnClient.submitApplication(appContext));
  LOG.info("Got new appId after submission : {}", getAppId());
} catch (YarnException yre) {
  this.setAppId(this.yarnClient.submitApplication(appContext));
  LOG.info("Got new appId after submission : {}", getAppId());

代码示例来源:origin: ShifuML/guagua

LOG.info("Submitting application to ASM");
  this.setAppId(this.yarnClient.submitApplication(appContext));
  LOG.info("Got new appId after submission : {}", getAppId());
} catch (YarnException yre) {
  this.setAppId(this.yarnClient.submitApplication(appContext));
  LOG.info("Got new appId after submission : {}", getAppId());

代码示例来源:origin: hortonworks/simple-yarn-app

yarnClient.submitApplication(appContext);

代码示例来源:origin: org.apache.gobblin/gobblin-yarn

this.yarnClient.submitApplication(appSubmissionContext);

代码示例来源:origin: com.cloudera.llama/llama

return rmClient.submitApplication(appContext);
} catch (Exception ex) {
 throw new LlamaException(ex, ErrorCode.AM_CANNOT_CREATE, queue);

27 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com