scala - spark 检索超过 20 条记录

Question

我有一个名为 students.json 的文件，如下所示:

{"uid":1,"name":"Michael","sid":1}
{"uid":2,"name":"Andy", "age":30,"sid":1}
{"uid":3,"name":"Jsaxsustin", "age":19,"sid":1}
{"uid":4,"name":"Andasxsay", "age":30,"sid":2}
{"uid":5,"name":"Jusewedtin", "age":19,"sid":1}
{"uid":6,"name":"Anwereddy", "age":30,"sid":3}
{"uid":7,"name":"Justdwedwein", "age":19,"sid":2}
{"uid":8,"name":"Andwedewy", "age":30,"sid":1}
{"uid":9,"name":"Justidedwn", "age":19,"sid":1}
{"uid":10,"name":"Anddwdey", "age":30,"sid":3}
{"uid":11,"name":"Michael","sid":1}
{"uid":12,"name":"Andy", "age":30,"sid":1}
{"uid":13,"name":"Jsaxsustin", "age":19,"sid":1}
{"uid":14,"name":"Andasxsay", "age":30,"sid":2}
{"uid":15,"name":"Jusewedtin", "age":19,"sid":1}
{"uid":16,"name":"Anwereddy", "age":30,"sid":3}
{"uid":17,"name":"Justdwedwein", "age":19,"sid":2}
{"uid":18,"name":"Andwe2fr3fdewy", "age":30,"sid":1}
{"uid":19,"name":"Justide4y45y54dwn", "age":19,"sid":1}
{"uid":20,"name":"Anddwd45y45yey", "age":30,"sid":3}
{"uid":21,"name":"Justdw45y45yedin", "age":19,"sid":1}
{"uid":22,"name":"An45y45ydy", "age":30,"sid":1}
{"uid":23,"name":"Jsaxsus4y54ytin", "age":19,"sid":1}
{"uid":24,"name":"Andas45y4y5xsay", "age":30,"sid":2}
{"uid":25,"name":"Jusewe4y5dtin", "age":19,"sid":1}
{"uid":26,"name":"Anwere45y45yddy", "age":30,"sid":3}
{"uid":27,"name":"Justdwe4y4y5dwein", "age":19,"sid":2}
{"uid":28,"name":"Andwede45ywy", "age":30,"sid":1}
{"uid":29,"name":"Justided45y45wn", "age":19,"sid":1}
{"uid":30,"name":"Anddwde4t4y", "age":30,"sid":3}
{"uid":31,"name":"Mich4y554ael","sid":1}
{"uid":32,"name":"An45ydy", "age":30,"sid":1}
{"uid":33,"name":"Jsaxsudfsstin", "age":19,"sid":1}
{"uid":34,"name":"Andasxssdfdsay", "age":30,"sid":2}
{"uid":35,"name":"Jusewedtsdfdsin", "age":19,"sid":1}
{"uid":36,"name":"Anweredsfdsdy", "age":30,"sid":3}
{"uid":37,"name":"Justdwedfsdwein", "age":19,"sid":2}
{"uid":38,"name":"Andwedewy", "age":30,"sid":1}
{"uid":39,"name":"Jdsfdsfustidedwn", "age":19,"sid":1}
{"uid":40,"name":"Ansdfdsey", "age":30,"sid":3}
{"uid":41,"name":"Jussdsdtdwedin", "age":19,"sid":1}

然后我使用 spark 导入数据并使用 sqlContext 对其进行 sql 查询。要检索我正在使用以下 scala 代码的信息(几乎是从 scala 上的 spark 示例中取出的):

val sqlContext = new org.apache.spark.sql.SQLContext(sc);
import sqlContext.implicits._
val students = sqlContext.read.json("../myTests/students.json");
students.registerTempTable("students");
sqlContext.sql("SELECT * FROM students").show;

这不是返回列表的所有项目，而是只返回 20 条记录(前 20 条记录)...如果在最后一行而不是

sqlContext.sql("SELECT * FROM students").show;

我放

sqlContext.sql("SELECT * FROM students where uid>10").show;

它给了我从 10 到 30 的 20 条记录。总之我想看20多条记录，怎么办？是否可以？我查看了文档，但没有阅读任何关于此的内容……我知道这可能有点傻，但在我进入更严重的事情之前我正在做一些测试……我下载了 spark 并正在运行它作为独立的只是为了看到 scala ...

我的 spark 初始化显示了这个(只有错误，我更改了配置，因此它不会显示信息)

15/07/17 11:37:55 ERROR Shell: Failed to locate the winutils binary in the hadoo
p binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Ha
doop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
        at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)

        at org.apache.hadoop.security.Groups.<init>(Groups.java:86)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group
s.java:280)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI
nformation.java:271)
        at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use
rGroupInformation.java:248)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(
UserGroupInformation.java:763)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou
pInformation.java:748)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr
oupInformation.java:621)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils
.scala:2162)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils
.scala:2162)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2162)
        at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:212)
        at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
        at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkIL
oop.scala:187)
        at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:2
17)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:949)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$Spark
ILoop$$process$1.apply(SparkILoop.scala:945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClass
Loader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$pr
ocess(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSub
mit$$runMain(SparkSubmit.scala:664)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:16
9)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) Client VM, Java 1.7.0_21)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
SQL context available as sqlContext.

scala> studentsstudentsstudentsstudents

谢谢

Answer 1

假设您想查看 100 行，您也可以使用，

df.show(100)

sqlContext.sql("SELECT * FROM students where uid>10").show(100)

链接到 method