- android - RelativeLayout 背景可绘制重叠内容
- android - 如何链接 cpufeatures lib 以获取 native android 库?
- java - OnItemClickListener 不起作用,但 OnLongItemClickListener 在自定义 ListView 中起作用
- java - Android 文件转字符串
我在启动 spark-sql shell 时收到以下错误。
但是当我使用它运行的命令启动 shell 时
./spark-sql --jars /usr/local/hive/lib/mysql-connector-java.jar
但是当我使用下面的命令以相同的方式启动 thrift 服务器时,它再次抛出相同的错误。
/usr/local/spark/sbin/start-thriftserver.sh --jars /usr/local/hive/lib/mysql-connector-java.jar
请帮助我理解如何解决这个问题,这样我就不必从外部传递 jar 路径,以及为什么它适用于 spark-sql 情况而不适用于 thrift 服务器。我是否需要在缺少的地方设置类路径?
如果您还需要什么,请告诉我。
5/10/18 05:15:33 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/10/18 05:15:33 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:47703
15/10/18 05:15:33 INFO util.Utils: Successfully started service 'HTTP file server' on port 47703.
15/10/18 05:15:33 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/10/18 05:15:38 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/10/18 05:15:38 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/10/18 05:15:38 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
15/10/18 05:15:38 INFO ui.SparkUI: Started SparkUI at http://192.168.1.12:4040
15/10/18 05:15:38 INFO spark.SparkContext: Added JAR file:/usr/local/hive/lib/mysql-connector-java.jar at http://192.168.1.12:47703/jars/mysql-connector-java.jar with timestamp 1445125538564
15/10/18 05:15:38 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@192.168.1.12:7077/user/Master...
15/10/18 05:15:38 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20151018051538-0018
15/10/18 05:15:38 INFO client.AppClient$ClientActor: Executor added: app-20151018051538-0018/0 on worker-20151018024224-192.168.1.12-50211 (192.168.1.12:50211) with 4 cores
15/10/18 05:15:38 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20151018051538-0018/0 on hostPort 192.168.1.12:50211 with 4 cores, 512.0 MB RAM
15/10/18 05:15:38 INFO client.AppClient$ClientActor: Executor updated: app-20151018051538-0018/0 is now LOADING
15/10/18 05:15:38 INFO client.AppClient$ClientActor: Executor updated: app-20151018051538-0018/0 is now RUNNING
15/10/18 05:15:39 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43460.
15/10/18 05:15:39 INFO netty.NettyBlockTransferService: Server created on 43460
15/10/18 05:15:39 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/10/18 05:15:39 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.12:43460 with 265.4 MB RAM, BlockManagerId(driver, 192.168.1.12, 43460)
15/10/18 05:15:39 INFO storage.BlockManagerMaster: Registered BlockManager
15/10/18 05:15:39 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
15/10/18 05:15:40 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
15/10/18 05:15:40 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead
15/10/18 05:15:40 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/10/18 05:15:40 INFO metastore.ObjectStore: ObjectStore, initialize called
15/10/18 05:15:41 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/10/18 05:15:41 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/10/18 05:15:41 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/10/18 05:15:41 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/10/18 05:15:42 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.1.12:56227/user/Executor#-1120183734]) with ID 0
15/10/18 05:15:42 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.12:34713 with 265.4 MB RAM, BlockManagerId(0, 192.168.1.12, 34713)
15/10/18 05:15:52 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead
15/10/18 05:15:52 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/10/18 05:15:52 INFO metastore.MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
15/10/18 05:15:54 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/10/18 05:15:54 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/10/18 05:16:01 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/10/18 05:16:01 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/10/18 05:16:03 INFO metastore.ObjectStore: Initialized ObjectStore
15/10/18 05:16:04 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/10/18 05:16:05 INFO metastore.HiveMetaStore: Added admin role in metastore
15/10/18 05:16:05 INFO metastore.HiveMetaStore: Added public role in metastore
15/10/18 05:16:05 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
15/10/18 05:16:05 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/10/18 05:16:05 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead
15/10/18 05:16:05 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes.
15/10/18 05:16:06 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead
15/10/18 05:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/10/18 05:16:06 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/10/18 05:16:06 INFO metastore.ObjectStore: ObjectStore, initialize called
15/10/18 05:16:07 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/10/18 05:16:07 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/10/18 05:16:07 WARN DataNucleus.Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:105)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:170)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:166)
at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:212)
at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:175)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:55)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:73)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
... 21 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
... 26 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
... 31 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:282)
at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:240)
at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:286)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
... 60 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "dbcp-builtin" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:259)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:131)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:85)
... 78 more
Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
at org.datanucleus.store.rdbms.connectionpool.DBCPBuiltinConnectionPoolFactory.createConnectionPool(DBCPBuiltinConnectionPoolFactory.java:49)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)
... 80 more
15/10/18 05:16:07 INFO spark.SparkContext: Invoking stop() from shutdown hook
最佳答案
将 mysql-connector-java-5.1.38-bin.jar 复制到 spark 2.x 版本中的 spark jars 位置
$ cp -r $HIVE_HOME/lib/mysql-connector-java-5.1.38-bin.jar $SPARK_HOME/jars/
关于mysql - 启动 spark sql 和 thrift server 时在类路径上找不到 com.mysql.jdbc.Driver,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/33192886/
本周我将在 Windows Server 2008 上设置一个专用的 SQL Server 2005 机器,并希望将其精简为尽可能简单,同时仍能发挥全部功能。 为此,“服务器核心”选项听起来很有吸引力
已关闭。此问题不符合Stack Overflow guidelines 。目前不接受答案。 这个问题似乎与 help center 中定义的范围内的编程无关。 . 已关闭 8 年前。 Improve
我获取了 2014 版本数据库的备份,并尝试在另一台服务器中将其恢复到具有相同名称和登录名的数据库中。此 SQL Server 版本是 2016。 恢复备份文件时,出现此错误: TITLE: Micr
我获取了 2014 版本数据库的备份,并尝试在另一台服务器中将其恢复到具有相同名称和登录名的数据库中。此 SQL Server 版本是 2016。 恢复备份文件时,出现此错误: TITLE: Micr
TFS 是否提供任何增强的方法来存储对 sql server 数据库所做的更改,而不是使用它来对在数据库上执行的 sql 语句的文本文件进行版本控制? 或者我正在寻找的功能是否仅在第 3 方工具(如
关闭。这个问题需要多问focused 。目前不接受答案。 想要改进此问题吗?更新问题,使其仅关注一个问题 editing this post . 已关闭 9 年前。 Improve this ques
我即将将我的 SQL Server 2012 实例升级到 SQL Server 2014。 我已经克隆了主机 Windows VM 并将其重命名为 foo-2012至 foo-2014 . 重新启动时
我想为 SQL Server 登录授予对数据库的访问权限。我知道 sp_grantdbaccess,但它已被弃用。我可以改用什么以及如何检查登录名是否还没有访问数据库的权限? 场景:UserA 创建数
客户别无选择,只能在接下来的几天内从 sql server 2000 迁移到 2008。测试显示 2005 年的重要功能出现了 Not Acceptable 性能下降,但 2008 年却没有。好消息是
我有一个测试数据库,我需要将其导出到我们客户的测试环境中。 这将是一次性的工作。 我正在使用 SQL Server 2005(我的测试数据库是 SQL Server 2005 Express) 执行此
我需要将一个 CSV 文件导入到 mongoDB 不幸的是我遇到了以下错误: error connecting to host: could not connect to server: se
我以为 R2 是一个补丁/服务包。我一直在寻找下载,但没有看到。因此,我假设 R2 是一个新版本,并且我需要 sqlserver 2008 r2 的安装介质来进行升级? 另外,我需要为新许可证付费吗?
我无法使用 SQL Server Management Studio 连接到 SQL Server。 我有一个连接字符串: 我尝试通过在服务器名中输入 myIP、在登录名中输入 MyID、在密码中
我们希望使用 SQL Server 加密来加密数据库中的几个列。我们还需要在生产和测试环境之间传输数据。看来最好的解决方案是在生产和测试服务器上使用相同的主 key 、证书和对称 key ,以便我可以
有没有可以分析 SQL Server 数据库潜在问题的工具? 例如: a foreign key column that is not indexed 没有 FILL FACTOR 的 uniquei
我正在尝试从我的 SQL 2012 BI 版本建立复制,但我收到一条奇怪的错误消息! "You cannot create a publication from server 'X' because
如果您使用 SQL Server 身份验证 (2005),登录详细信息是否以明文形式通过网络发送? 最佳答案 如您所愿,安全无忧... 您可以相当轻松地配置 SSL,如果您没有受信任的证书,如果您强制
我想将数据从一个表复制到不同服务器之间的另一个表。 如果是在同一服务器和不同的数据库中,我使用了以下 SELECT * INTO DB1..TBL1 FROM DB2..TBL1 (to copy w
我希望得到一些帮助,因为我在这个问题上已经被困了 2 天了! 场景:我可以从我的开发计算机(和其他同事)连接到 SERVER\INSTANCE,但无法从另一个 SQL Server 连接。我得到的错误
我正在尝试从我的 SQL 2012 BI 版本建立复制,但我收到一条奇怪的错误消息! "You cannot create a publication from server 'X' because
我是一名优秀的程序员,十分优秀!