gpt4 book ai didi

org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore类的使用及代码示例

转载 作者:知者 更新时间:2024-03-17 00:47:31 25 4
gpt4 key购买 nike

本文整理了Java中org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore类的一些代码示例,展示了ZooKeeperSubmittedJobGraphStore类的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台,是从一些精选项目中提取出来的代码,具有较强的参考意义,能在一定程度帮忙到你。ZooKeeperSubmittedJobGraphStore类的具体详情如下:
包路径:org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore
类名称:ZooKeeperSubmittedJobGraphStore

ZooKeeperSubmittedJobGraphStore介绍

[英]SubmittedJobGraph instances for JobManagers running in HighAvailabilityMode#ZOOKEEPER.

Each job graph creates ZNode:

+----O /flink/jobgraphs/<job-id> 1 [persistent] 
. 
. 
. 
+----O /flink/jobgraphs/<job-id> N [persistent]

The root path is watched to detect concurrent modifications in corner situations where multiple instances operate concurrently. The job manager acts as a SubmittedJobGraphListenerto react to such situations.
[中]在HighAvailabilityMode#ZOOKEEPER中运行的JobManager的SubmittedJobGraph实例。
每个作业图都会创建ZNode:

+----O /flink/jobgraphs/<job-id> 1 [persistent] 
. 
. 
. 
+----O /flink/jobgraphs/<job-id> N [persistent]

在多个实例同时操作的情况下,监视根路径以检测并发修改。工作经理充当提交人,对此类情况做出反应。

代码示例

代码示例来源:origin: org.apache.flink/flink-runtime_2.10

@Override
public void removeJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  String path = getPathForJob(jobId);
  LOG.debug("Removing job graph {} from {}{}.", jobId, zooKeeperFullBasePath, path);
  synchronized (cacheLock) {
    if (addedJobGraphs.contains(jobId)) {
      jobGraphsInZooKeeper.releaseAndTryRemove(path);
      addedJobGraphs.remove(jobId);
    }
  }
  LOG.info("Removed job graph {} from ZooKeeper.", jobId);
}

代码示例来源:origin: org.apache.flink/flink-runtime_2.11

@Override
public Collection<JobID> getJobIds() throws Exception {
  Collection<String> paths;
  LOG.debug("Retrieving all stored job ids from ZooKeeper under {}.", zooKeeperFullBasePath);
  try {
    paths = jobGraphsInZooKeeper.getAllPaths();
  } catch (Exception e) {
    throw new Exception("Failed to retrieve entry paths from ZooKeeperStateHandleStore.", e);
  }
  List<JobID> jobIds = new ArrayList<>(paths.size());
  for (String path : paths) {
    try {
      jobIds.add(jobIdfromPath(path));
    } catch (Exception exception) {
      LOG.warn("Could not parse job id from {}. This indicates a malformed path.", path, exception);
    }
  }
  return jobIds;
}

代码示例来源:origin: org.apache.flink/flink-runtime

@Override
public void putJobGraph(SubmittedJobGraph jobGraph) throws Exception {
  checkNotNull(jobGraph, "Job graph");
  String path = getPathForJob(jobGraph.getJobId());
      verifyIsRunning();

代码示例来源:origin: org.apache.flink/flink-runtime_2.11

/**
 * Creates a {@link ZooKeeperSubmittedJobGraphStore} instance.
 *
 * @param client        The {@link CuratorFramework} ZooKeeper client to use
 * @param configuration {@link Configuration} object
 * @return {@link ZooKeeperSubmittedJobGraphStore} instance
 * @throws Exception if the submitted job graph store cannot be created
 */
public static ZooKeeperSubmittedJobGraphStore createSubmittedJobGraphs(
    CuratorFramework client,
    Configuration configuration) throws Exception {
  checkNotNull(configuration, "Configuration");
  RetrievableStateStorageHelper<SubmittedJobGraph> stateStorage = createFileSystemStateStorage(configuration, "submittedJobGraph");
  // ZooKeeper submitted jobs root dir
  String zooKeeperSubmittedJobsPath = configuration.getString(HighAvailabilityOptions.HA_ZOOKEEPER_JOBGRAPHS_PATH);
  return new ZooKeeperSubmittedJobGraphStore(
    client,
    zooKeeperSubmittedJobsPath,
    stateStorage);
}

代码示例来源:origin: com.alibaba.blink/flink-runtime

@Override
public void putJobGraph(SubmittedJobGraph jobGraph) throws Exception {
  checkNotNull(jobGraph, "Job graph");
  String path = getPathForJob(jobGraph.getJobId());
      verifyIsRunning();

代码示例来源:origin: com.alibaba.blink/flink-runtime

/**
 * Creates a {@link ZooKeeperSubmittedJobGraphStore} instance.
 *
 * @param client        The {@link CuratorFramework} ZooKeeper client to use
 * @param configuration {@link Configuration} object
 * @param executor to run ZooKeeper callbacks
 * @return {@link ZooKeeperSubmittedJobGraphStore} instance
 * @throws Exception if the submitted job graph store cannot be created
 */
public static ZooKeeperSubmittedJobGraphStore createSubmittedJobGraphs(
    CuratorFramework client,
    Configuration configuration,
    Executor executor) throws Exception {
  checkNotNull(configuration, "Configuration");
  RetrievableStateStorageHelper<SubmittedJobGraph> stateStorage = createFileSystemStateStorage(configuration, "submittedJobGraph");
  // ZooKeeper submitted jobs root dir
  String zooKeeperSubmittedJobsPath = configuration.getString(HighAvailabilityOptions.HA_ZOOKEEPER_JOBGRAPHS_PATH);
  return new ZooKeeperSubmittedJobGraphStore(
      client, zooKeeperSubmittedJobsPath, stateStorage, executor);
}

代码示例来源:origin: org.apache.flink/flink-runtime_2.11

@Override
public void putJobGraph(SubmittedJobGraph jobGraph) throws Exception {
  checkNotNull(jobGraph, "Job graph");
  String path = getPathForJob(jobGraph.getJobId());
      verifyIsRunning();

代码示例来源:origin: com.alibaba.blink/flink-runtime

@Override
public void removeJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  String path = getPathForJob(jobId);
  LOG.debug("Removing job graph {} from {}{}.", jobId, zooKeeperFullBasePath, path);
  synchronized (cacheLock) {
    if (addedJobGraphs.contains(jobId)) {
      jobGraphsInZooKeeper.releaseAndTryRemove(path);
      addedJobGraphs.remove(jobId);
    }
  }
  LOG.info("Removed job graph {} from ZooKeeper.", jobId);
}

代码示例来源:origin: org.apache.flink/flink-runtime_2.10

@Override
public Collection<JobID> getJobIds() throws Exception {
  Collection<String> paths;
  LOG.debug("Retrieving all stored job ids from ZooKeeper under {}.", zooKeeperFullBasePath);
  try {
    paths = jobGraphsInZooKeeper.getAllPaths();
  } catch (Exception e) {
    throw new Exception("Failed to retrieve entry paths from ZooKeeperStateHandleStore.", e);
  }
  List<JobID> jobIds = new ArrayList<>(paths.size());
  for (String path : paths) {
    try {
      jobIds.add(jobIdfromPath(path));
    } catch (Exception exception) {
      LOG.warn("Could not parse job id from {}. This indicates a malformed path.", path, exception);
    }
  }
  return jobIds;
}

代码示例来源:origin: org.apache.flink/flink-runtime

/**
 * Creates a {@link ZooKeeperSubmittedJobGraphStore} instance.
 *
 * @param client        The {@link CuratorFramework} ZooKeeper client to use
 * @param configuration {@link Configuration} object
 * @return {@link ZooKeeperSubmittedJobGraphStore} instance
 * @throws Exception if the submitted job graph store cannot be created
 */
public static ZooKeeperSubmittedJobGraphStore createSubmittedJobGraphs(
    CuratorFramework client,
    Configuration configuration) throws Exception {
  checkNotNull(configuration, "Configuration");
  RetrievableStateStorageHelper<SubmittedJobGraph> stateStorage = createFileSystemStateStorage(configuration, "submittedJobGraph");
  // ZooKeeper submitted jobs root dir
  String zooKeeperSubmittedJobsPath = configuration.getString(HighAvailabilityOptions.HA_ZOOKEEPER_JOBGRAPHS_PATH);
  return new ZooKeeperSubmittedJobGraphStore(
    client,
    zooKeeperSubmittedJobsPath,
    stateStorage);
}

代码示例来源:origin: org.apache.flink/flink-runtime_2.10

@Override
public void putJobGraph(SubmittedJobGraph jobGraph) throws Exception {
  checkNotNull(jobGraph, "Job graph");
  String path = getPathForJob(jobGraph.getJobId());
      verifyIsRunning();

代码示例来源:origin: org.apache.flink/flink-runtime

@Override
public void releaseJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  final String path = getPathForJob(jobId);
  LOG.debug("Releasing locks of job graph {} from {}{}.", jobId, zooKeeperFullBasePath, path);
  synchronized (cacheLock) {
    if (addedJobGraphs.contains(jobId)) {
      jobGraphsInZooKeeper.release(path);
      addedJobGraphs.remove(jobId);
    }
  }
  LOG.info("Released locks of job graph {} from ZooKeeper.", jobId);
}

代码示例来源:origin: org.apache.flink/flink-runtime

@Override
public Collection<JobID> getJobIds() throws Exception {
  Collection<String> paths;
  LOG.debug("Retrieving all stored job ids from ZooKeeper under {}.", zooKeeperFullBasePath);
  try {
    paths = jobGraphsInZooKeeper.getAllPaths();
  } catch (Exception e) {
    throw new Exception("Failed to retrieve entry paths from ZooKeeperStateHandleStore.", e);
  }
  List<JobID> jobIds = new ArrayList<>(paths.size());
  for (String path : paths) {
    try {
      jobIds.add(jobIdfromPath(path));
    } catch (Exception exception) {
      LOG.warn("Could not parse job id from {}. This indicates a malformed path.", path, exception);
    }
  }
  return jobIds;
}

代码示例来源:origin: org.apache.flink/flink-runtime_2.10

/**
 * Creates a {@link ZooKeeperSubmittedJobGraphStore} instance.
 *
 * @param client        The {@link CuratorFramework} ZooKeeper client to use
 * @param configuration {@link Configuration} object
 * @param executor to run ZooKeeper callbacks
 * @return {@link ZooKeeperSubmittedJobGraphStore} instance
 * @throws Exception if the submitted job graph store cannot be created
 */
public static ZooKeeperSubmittedJobGraphStore createSubmittedJobGraphs(
    CuratorFramework client,
    Configuration configuration,
    Executor executor) throws Exception {
  checkNotNull(configuration, "Configuration");
  RetrievableStateStorageHelper<SubmittedJobGraph> stateStorage = createFileSystemStateStorage(configuration, "submittedJobGraph");
  // ZooKeeper submitted jobs root dir
  String zooKeeperSubmittedJobsPath = ConfigurationUtil.getStringWithDeprecatedKeys(
      configuration,
      ConfigConstants.HA_ZOOKEEPER_JOBGRAPHS_PATH,
      ConfigConstants.DEFAULT_ZOOKEEPER_JOBGRAPHS_PATH,
      ConfigConstants.ZOOKEEPER_JOBGRAPHS_PATH);
  return new ZooKeeperSubmittedJobGraphStore(
      client, zooKeeperSubmittedJobsPath, stateStorage, executor);
}

代码示例来源:origin: org.apache.flink/flink-runtime_2.10

@Override
public SubmittedJobGraph recoverJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  final String path = getPathForJob(jobId);
    verifyIsRunning();

代码示例来源:origin: org.apache.flink/flink-runtime_2.11

@Override
public void releaseJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  final String path = getPathForJob(jobId);
  LOG.debug("Releasing locks of job graph {} from {}{}.", jobId, zooKeeperFullBasePath, path);
  synchronized (cacheLock) {
    if (addedJobGraphs.contains(jobId)) {
      jobGraphsInZooKeeper.release(path);
      addedJobGraphs.remove(jobId);
    }
  }
  LOG.info("Released locks of job graph {} from ZooKeeper.", jobId);
}

代码示例来源:origin: com.alibaba.blink/flink-runtime

@Override
public Collection<JobID> getJobIds() throws Exception {
  Collection<String> paths;
  LOG.debug("Retrieving all stored job ids from ZooKeeper under {}.", zooKeeperFullBasePath);
  try {
    paths = jobGraphsInZooKeeper.getAllPaths();
  } catch (Exception e) {
    throw new Exception("Failed to retrieve entry paths from ZooKeeperStateHandleStore.", e);
  }
  List<JobID> jobIds = new ArrayList<>(paths.size());
  for (String path : paths) {
    try {
      jobIds.add(jobIdfromPath(path));
    } catch (Exception exception) {
      LOG.warn("Could not parse job id from {}. This indicates a malformed path.", path, exception);
    }
  }
  return jobIds;
}

代码示例来源:origin: com.alibaba.blink/flink-runtime

public SubmittedJobGraph recoverJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  final String path = getPathForJob(jobId);
    verifyIsRunning();

代码示例来源:origin: org.apache.flink/flink-runtime_2.11

@Override
public void removeJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  String path = getPathForJob(jobId);
  LOG.debug("Removing job graph {} from {}{}.", jobId, zooKeeperFullBasePath, path);
  synchronized (cacheLock) {
    if (addedJobGraphs.contains(jobId)) {
      if (jobGraphsInZooKeeper.releaseAndTryRemove(path)) {
        addedJobGraphs.remove(jobId);
      } else {
        throw new FlinkException(String.format("Could not remove job graph with job id %s from ZooKeeper.", jobId));
      }
    }
  }
  LOG.info("Removed job graph {} from ZooKeeper.", jobId);
}

代码示例来源:origin: org.apache.flink/flink-runtime_2.11

public SubmittedJobGraph recoverJobGraph(JobID jobId) throws Exception {
  checkNotNull(jobId, "Job ID");
  final String path = getPathForJob(jobId);
    verifyIsRunning();

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com