amazon-ec2 - 如何使用Amazon AMI在Amazon AWS EC2或EMR上安装GUI

Question

我需要运行需要GUI界面才能启动和配置的应用程序。我还需要能够在Amazon的EC2服务和EMR服务上运行此应用程序。 EMR要求意味着它必须在Amazon的Linux AMI上运行。

经过广泛的搜索，我一直找不到任何现成的解决方案，特别是在Amazon AMI上运行的要求。最接近且最常用的解决方案是here。不幸的是，它是在RHEL6实例上开发的，该实例与Amazon的AMI完全不同，因此该解决方案无法正常工作。

我在下面发布我的解决方案。希望它可以从为找到正确配方而进行的大量实验中省掉一些其他的东西。

Answer 1

这是使GUI在Amazon AMI上运行的解决方案。我以post作为起点，但必须进行许多更改才能使其在Amazon AMI上运行。我还添加了其他信息，以使这种工作以合理的自动化方式完成，因此需要多次提出这种环境的个人可以轻松完成这项工作。

注意：本文中包含很多评论。我事先表示歉意，但我认为这对需要进行修饰的人可能会有所帮助，如果他们能够理解为什么在此过程中做出了各种选择。

下面包含的脚本会在安装过程中安装一些文件。有关这些脚本使用的文件和目录结构的列表，请参见第4节。

步骤1.安装桌面

执行“ yum更新”后，大多数解决方案包括

sudo yum groupinstall -y "Desktop"

这个看似简单的步骤需要在Amazon AMI上花费更多的精力。此组未在Amazon AMI中配置（从此处开始为AAMI）。默认情况下，AAMI已安装并启用了亚马逊自己的存储库。还安装了epel存储库，但是默认情况下它被禁用。启用epel之后，我找到了Desktop组，但其中未填充软件包。我还找到了Xfce（另一种台式机替代品）。最终，我决定安装Xfce，而不是台式机。尽管如此，这并非直截了当，但最终导致了解决方案。

在这里值得注意的是，我首先尝试的是安装centos存储库并从那里安装Desktop组。最初，这似乎很有希望。该小组已满员。但是，经过一番努力，我最终决定依赖项和AAMI上已经安装的软件包之间存在太多版本冲突。

这使我从epel仓库中选择了Xfce。由于epel存储库已经安装在AAMI上，因此我认为与Amazon存储库之间的依赖项版本协调会更好。这通常是正确的。在epel仓库或Amazon仓库中发现了许多依赖项。对于那些不是的，我可以在centos存储库中找到它们，并且在大多数情况下，它们是叶依赖项。因此，大多数麻烦来自于centos回购中的少数依赖项，这些子依赖项与亚马逊或epel回购有冲突。最后，需要一些技巧来绕过依赖关系冲突。我试图尽量减少这些。这是安装Xfce的脚本

installGui.sh

#!/bin/bash

# echo each command
set -x

# assumes RSRC_DIR and IS_EMR set by parent script
YUM_RSRC_DIR=$RSRC_DIR/yum

sudo yum -y update

# Most info I've found on installing a GUI on AWS suggests to install using
#> sudo yum groupinstall -y "Desktop"
# This group is not available by default on the Amazon Linux AMI.  The group
# is listed if the epel repo is enabled, but it is empty.  I tried installing
# the centos repo, which does have support for this group, but it simply end
# up having to many dependency version conflicts with packages already installed
# by the Amazon repos.
#
# I found the path of least resistance to be installing the group Xfce from
# the epel repo. The epel repo is already included in amazon image, just not enabled.
# So I'm guessing there was at least some consideration by Amazon to align
# the dependency versions of this repo with the Amazon repos.
#
# My general approach to this problem was to start with the last command:
#> sudo yum groupinstall -y Xfce
# which will generate a list of missing dependencies.  The script below
# essentially works backwards through that list to eliminate all the
# missing dependencies.
#
# In general, many of the dependencies required by Xfce are found in either
# the epel repo or the Amazon repos.  Most of the remaining dependencies can be
# found in the centos repo, and either don't have any further dependencies, or if they
# do those dependencies are satisfied with the centos repo with no collisions
# in the epel or amazon repo.  Then there are a couple of oddball dependencies
# to clean up.

# if yum-config-manager is not found then install yum-utils
#> sudo yum install yum-utils
sudo yum-config-manager --enable epel

# install centos repo
# place the repo config @  /etc/yum.repos.d/centos.repo
sudo cp $YUM_RSRC_DIR/yum.repos.d/centos.repo /etc/yum.repos.d/

# The config centos.repo specifies the key with a URL.  If for some reason the key
# must be in a local file, it can be found here: https://www.centos.org/keys/RPM-GPG-KEY-CentOS-6
# It can be installed to the right location in one step:
#> wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 https://www.centos.org/keys/RPM-GPG-KEY-CentOS-6
# Note, a key file must also be installed in the system key ring.  The docs are a bit confusing
# on this, I found that I needed to run both gpg AND then followed by rpm, eg:
#> sudo gpg --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
#> sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

# I found there are a lot of version conflicts between the centos, Amazon and epel repos.
# So I did not enable the centos repo generally.  Instead I used the --enablerepo switch
# enable it explicitly for each yum command that required it.  This only works for yum.  If
# rpm must be used, then yum-config-manager must be used to enable/disable repos as a
# separate step.
#
# Another problem I ran into was yum installing the 32-bit (*.i686) package rather than
# the 64-bit (*.x86_64) verision of the package.  I never figured out why.  So I had
# to specify the *.x86_64 package explicitly.  The search tools (eg. 'whatprovides')
# did not list the 64 bit package either even though a manual search through the
# package showed the 64 bit components were present.
#
# Sometimes it is difficult to determine which package must be in installed to satisfy
# a particular dependency.  'whatprovides' is a very useful tool for this
#> yum --enablerepo centos whatprovides libgdk_pixbuf-2.0.so.0
#> rpm -q --whatprovides libgdk_pixbuf

sudo yum --enablerepo centos install -y gdk-pixbuf2.x86_64
sudo yum --enablerepo centos install -y gtk2.x86_64
sudo yum --enablerepo centos install -y libnotify.x86_64
sudo yum --enablerepo centos install -y gnome-icon-theme
sudo yum --enablerepo centos install -y redhat-menus
sudo yum --enablerepo centos install -y gstreamer-plugins-base.x86_64

# problem when we get to libvte, installing libvte requires expat, which conflicts with amazon lib
# the centos package version was older and did not install right lib version
# but … the expat dependency was coming from a dependency on python-libs.
# the easiest workaround was to install python using the amazon repo, that in turn
# installs a version of python libs that is compatible with the version of libexpat on the system.

sudo yum install -y python
sudo yum --enablerepo centos install -y vte.x86_64

sudo yum --enablerepo centos install -y libical.x86_64
sudo yum --enablerepo centos install -y gnome-keyring.x86_64

# another sticky point, xfdesktop requires desktop-backgrounds-basic, but ‘whatprovides’ does not 
# provide any packages for this query (not sure why).  It turns out this is provided by the centos 
# repo, installing ‘desktop-backgrounds-basic’ will try to install the package redhat-logos, but 
# unfortunately this is obsoleted by Amazon’s generic-logos package
# The only way I could find to get around this was to erase the generic logos package.
# This doesn't seem too risky since this is just images for the desktop and menus.
#
sudo yum erase -y generic-logos

# Amazon repo must be disabled to prevent interference with the install
# of redhat-logos
sudo yum --disablerepo amzn-main --enablerepo centos install -y redhat-logos

# next problem is a dependency on dbus.  The dependency comes from dbus-x11 in 
# centos repo.  It requires dbus version 1.2.24, the amazon image already has
# version 1.6.12 installed.  Since the dbus-x11 is only used by the GUI package,
# easiest way around this is to install dbus-x11 with no dependency checks.
# So it will use the newer version of dbus (should be OK).  The main thing that could be a problem
# here is if it skips some other dependency.  When doing manually, its possible to run the install until
# the only error left is the dbus dependency.  It’s a bit risky running in a script since, basically it’s assuming
# all the dependencies are already in place.
yumdownloader --enablerepo centos dbus-x11.x86_64
sudo rpm -ivh --nodeps dbus-x11-1.2.24-8.el6_6.x86_64.rpm
rm dbus-x11-1.2.24-8.el6_6.x86_64.rpm

sudo yum install -y xfdesktop.x86_64

# We need the version of poppler-glib from centos repo, but it is found in several repos.
# Disable the other repos for this step.
# On EMR systems a newer version of poppler is already installed.  So move up 1 level
# in dependency chain and force install of tumbler.

if [ $IS_EMR -eq 1 ]
then
    yumdownloader --enablerepo centos tumbler.x86_64
    sudo rpm -ivh --nodeps tumbler-0.1.21-1.el6.x86_64.rpm
else
    sudo yum --disablerepo amzn-main --disablerepo amzn-updates --disablerepo epel --enablerepo centos install -y poppler-glib
fi


sudo yum install  --enablerepo centos -y polkit-gnome.x86_64
sudo yum install  --enablerepo centos  -y control-center-filesystem.x86_64

sudo yum groupinstall -y Xfce

以下是centos存储库配置文件的内容：

centos.repo

[centos]
name=CentOS mirror
baseurl=http://repo1.ash.innoscale.net/centos/6/os/x86_64/
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=https://www.centos.org/keys/RPM-GPG-KEY-CentOS-6

如果您只需要在Amazon AMI上安装台式机软件包的秘诀，那么您就完成了。本文的其余部分介绍如何配置VNC以通过SSH隧道访问桌面，以及如何打包所有这些内容，以便可以从脚本轻松启动实例。

步骤2.安装和配置VNC

以下是我用于安装GUI的顶级脚本。配置完几个变量后，首先要做的是调用上述步骤1中的脚本。自从我将其构建为可在常规ec2实例或emr上以root或ec2-user身份运行以来，该脚本还有一些额外的负担。基本步骤是


安装libXfont
安装Tiger-vnc-server
安装VNC服务器配置文件
在用户主目录中创建一个.vnc目录
将xstartup文件安装在.vnc目录中
在.vnc目录中安装一个虚拟passwd文件
启动VNC服务器


需要注意的几个关键点：

假设您将通过SSH隧道访问VNC服务器。最后，这实际上似乎是最简单，最可靠的安全方法。由于您可能已在安全组规范中为SSH打开了端口，因此您无需对其进行任何更改。同样，VNC客户端/服务器的加密配置也不是直接的。似乎很容易犯错误，并使您的通信不加密。设置在vncservers文件中。 -localhost开关告诉vnc仅接受本地连接。 '-nolisten tcp'告诉关联的xserver模块也不接受来自网络的连接。最后，“-SecurityTypes None”开关使您无需输入密码即可打开VNC会话，因为进入计算机的唯一方法是通过ssh，所以额外的密码检查似乎是多余的。

xstartup文件确定首次启动VNC会话时将开始什么。我注意到有关此主题的许多帖子都忽略了这一点。如果您不告诉它启动Xfce桌面，则在启动VNC时只会得到一个空白窗口。我在这里的配置非常简单。

即使我在上面提到了VNC服务器被配置为不提示输入密码，它仍然需要.vnc目录中的passwd文件才能启动服务器。首次运行该脚本时，它将在尝试启动服务器时失败。通过ssh登录到计算机并运行'vncpasswd'。它将在.vnc目录中创建一个passwd文件，您可以将其保存以在安装过程中用作这些脚本的一部分。注意，我读过VNC并没有做任何复杂的事情来保护passwd文件。因此，我不建议您将密码用于其他更重要的帐户。

installGui.sh

#!/bin/bash

# echo each command
set -x

BIN_DIR="${BASH_SOURCE%/*}"
ROOT_DIR=$(dirname $BIN_DIR)
RSRC_DIR=$ROOT_DIR/rsrc
VNC_DIR=$RSRC_DIR/vnc

# Install user config files into ec2-user home directory
# if it is available.  In practice, this should always
# be true

if [ -d "/home/ec2-user" ]
then
   USER_ACCT=ec2-user
else
   USER_ACCT=hadoop
fi

HOME_DIR="/home"

# Use existence of hadoop home directory as proxy to determine if
# this is an EMR system.  Can be used later to differentiate
# steps on EC2 system vs EMR.
if [ -d "/home/hadoop" ]
then
    IS_EMR=1
else
    IS_EMR=0
fi


# execute Xfce desktop install
. "$BIN_DIR/installXfce.sh"

# now roughly follow the following from step 3: https://devopscube.com/setup-gui-for-amazon-ec2-linux/

sudo yum install -y pixman pixman-devel libXfont

sudo yum -y install tigervnc-server


# install the user account configuration file.
# This setup assumes the user will always connect to the VNC server
# through an SSH tunnel.  This is generally more secure, easier to
# configure and easier to get correct than trying to allow direct
# connections via TCP.
# Therefore, config VNC server to only accept local connections, and
# no password required.
sudo cp $VNC_DIR/vncservers-$USER_ACCT /etc/sysconfig/vncservers

# install the user account, vnc config files

sudo mkdir $HOME_DIR/$USER_ACCT/.vnc
sudo chown $USER_ACCT:$USER_ACCT $HOME_DIR/$USER_ACCT/.vnc

# need xstartup file to tell vncserver to start the window manager
sudo cp $VNC_DIR/xstartup $HOME_DIR/$USER_ACCT/.vnc/
sudo chown $USER_ACCT:$USER_ACCT $HOME_DIR/$USER_ACCT/.vnc/xstartup

# Even though the VNC server is config'd to not require a passwd, the
# server still looks for the passwd file when it starts the session.
# It will fail if the passwd file is not found.
# The first time these scripts are run, the final step will fail.
# Then manually run
#> vncpasswd
# It will create the file ~/.vnc/passwd.  Then save this file to persistent
# storage so that it can be installed to the user account during
# server initialization.

sudo cp $ROOT_DIR/home/user/.vnc/passwd $HOME_DIR/$USER_ACCT/.vnc/
sudo chown $USER_ACCT:$USER_ACCT $HOME_DIR/$USER_ACCT/.vnc/passwd

# This script will be running as root if called from the EC2 launch
# command.  VNC server needs to be started as the user that
# you will connect to the server as (eg. ec2-user, hadoop, etc.)
sudo su -c "sudo service vncserver start" -s /bin/sh $USER_ACCT

# how to stop vncserver
# vncserver -kill :1

# On the remote client
# 1. start the ssh tunner
#> ssh -i ~/.ssh/<YOUR_KEY_FILE>.pem -L 5901:localhost:5901 -N ec2-user@<YOUR_SERVER_PUBLIC_IP>
#    for debugging connection use -vvv switch
# 2. connect to the vnc server using client on the remote machine.  When
#    prompted for the IP address, use 'localhost:5901'
#    This connects to port 5901 on your local machine, which is where the ssh
#    tunnel is listening.

vncservers

# The VNCSERVERS variable is a list of display:user pairs.
#
# Uncomment the lines below to start a VNC server on display :2
# as my 'myusername' (adjust this to your own).  You will also
# need to set a VNC password; run 'man vncpasswd' to see how
# to do that.  
#
# DO NOT RUN THIS SERVICE if your local area network is
# untrusted!  For a secure way of using VNC, see this URL:
# http://kbase.redhat.com/faq/docs/DOC-7028

# Use "-nolisten tcp" to prevent X connections to your VNC server via TCP.

# Use "-localhost" to prevent remote VNC clients connecting except when
# doing so through a secure tunnel.  See the "-via" option in the
# `man vncviewer' manual page.

# Use "-SecurityTypes None" to allow session login without a password.
# This should only be used in combination with "-localhost"
# Note: VNC server still looks for the passwd file in ~/.vnc directory
# when the session starts regardless of whether the user is
# required to enter a passwd.

# VNCSERVERS="2:myusername"
# VNCSERVERARGS[2]="-geometry 800x600 -nolisten tcp -localhost"
VNCSERVERS="1:ec2-user"
VNCSERVERARGS[1]="-geometry 1280x1024 -nolisten tcp -localhost -SecurityTypes None"

xstartup

#!/bin/sh

unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
# exec /etc/X11/xinit/xinitrc
/usr/share/vte/termcap/xterm &
/usr/bin/startxfce4 &

步骤3.连接到您的实例

在EC2上运行VNC服务器后，您可以尝试连接到它。首先打开到您实例的SSH隧道。 5901是VNC服务器从vncservers文件监听显示1的端口。它将侦听端口5902上的显示2，等等。此命令创建从本地计算机上的端口5901到实例上的端口5901的隧道。

ssh -i ~/.ssh/<YOUR_KEY_FILE>.pem -L 5901:localhost:5901 -N ec2-user@<YOUR_SERVER_PUBLIC_IP>

现在打开您的首选VNC客户端。在提示输入服务器IP地址的地方输入：


本地主机：5901


如果什么都没有发生，则说明启动vnc服务器时出现问题，或者存在连接问题导致客户端无法访问服务器，或者vncservers配置文件中可能存在问题

如果出现一个窗口，但它只是空白，请检查Xfce安装是否成功完成以及xstartup文件是否已安装。

步骤4.简化

如果您只需要执行一次，则将脚本sftp'到您的实例并手动运行就可以了。否则，当您确实需要使用GUI启动实例时，您将要尽可能地使其自动化，以使其更快并且更不易出错。

自动化的第一步是创建一个EFS卷，其中包含在实例启动时可以挂载的脚本和配置文件。亚马逊在创建网络文件系统方面有很多 info。创建体积时要注意几点。如果您不希望将卷向世界公开，则可以创建一个自定义安全组以用于EFS卷。我为我的EFS卷创建了安全组（称为NFS_Mount），该安全组仅允许来自其他安全组之一的端口2049上的入站TCP通信，称为MasterVNC。然后，在创建实例时，请确保将MasterVNC安全组与此实例相关联。否则，EFS卷将不允许您的实例与其连接。

现在挂载EFS卷：

sudo mkdir /mnt/YOUR_MOUNT_POINT_DIR
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-YOUR_EFS_ID.efs.us-east-1.amazonaws.com:/ /mnt/YOUR_MOUNT_POINT_DIR

现在，使用以下目录结构，使用步骤1和2中提到的6个文件填充/ mnt / YOUR_MOUNT_POINT_DIR。回想一下，您必须在第一次使用命令'vncpasswd'创建passwd文件。它将在〜/ .vnc / passwd中创建文件。


/mnt/YOUR_MOUNT_POINT_DIR/bin/installGui.sh
/mnt/YOUR_MOUNT_POINT_DIR/bin/installXfce.sh

/ mnt / YOUR_MOUNT_POINT_DIR / rsrc / vnc / vncservers-ec2-user
/ mnt / YOUR_MOUNT_POINT_DIR / rsrc / vnc / xstartup
/ mnt / YOUR_MOUNT_POINT_DIR / rsrc / vnc / passwd

/mnt/YOUR_MOUNT_POINT_DIR/rsrc/yum/yum.repos.d/centos.repo


此时，使用GUI设置实例应该非常容易。照常创建实例（确保包括MasterVNC安全组），将ssh绑定到实例，安装EFS卷，然后运行installGui.sh脚本。

步骤5.自动化

您可以更进一步，并使用本地计算机上的AWS CLI工具一步一步启动实例。为此，您将需要挂载EFS卷并使用AWS CLI命令的参数运行installGui.sh脚本。这仅需要创建一个顶级脚本并将其传递给CLI命令。

当然会有一些并发症。 EC2和EMR使用不同的开关和机制来附加脚本。而且，在EMR上，我只希望将GUI安装在主节点（而不是核心或任务节点）上。

启动EC2实例需要使用--user-data开关将脚本嵌入命令中。通过在本地计算机上指定脚本文件的绝对路径，可以轻松完成此操作。

aws ec2 run-instances --user-data file:///PATH_TO_YOUR_SCRIPT/top.sh  ... other options

EMR启动不支持从本地文件嵌入脚本。相反，您可以在引导操作中指定S3 URI。

aws emr create-cluster --bootstrap-actions '[{"Path":"s3://YOUR_BUCKET/YOUR_DIR/top.sh","Name":"Custom action"}]' ... other options

最后，您将在top.sh中的大多数脚本下面看到确定该机器是基本EC2实例还是EMR主站的函数。如果不是这样，脚本可能是3行。您可能想知道为什么不只使用内置的“ run-if”引导操作而不是编写自己的函数。内置的“运行时”脚本存在错误，并且无法正确运行S3中的脚本。

一旦将它们放入init序列中，对其进行调试可能是一个挑战。日志文件可以帮助您一件事：/var/log/cloud-init-output.log。这将从引导程序初始化期间运行的脚本中捕获所有控制台输出。

top.sh

#!/bin/bash

# note: conditional bootstrap function run-if has a bug, workaround ...
# this function adapted from https://forums.aws.amazon.com/thread.jspa?threadID=222418
# Determine if we are running on the master node.
# 0 - running on master, or non EMR node
# 1 - running on a task or core node

check_if_master_or_non_emr() {
    python - <<'__SCRIPT__'
import sys
import json

instance_file = "/mnt/var/lib/info/instance.json"

try:
    with open(instance_file) as f:
        props = json.load(f)
    is_master_or_non_emr = props.get('isMaster', False)

except IOError as ex:
    is_master_or_non_emr = True   # file will not exist when testing on a non-emr machine

if is_master_or_non_emr:
    sys.exit(1)
else:
    sys.exit(0)
__SCRIPT__
}

check_if_master_or_non_emr
IS_MASTER_OR_NON_EMR=$?

# If this machine is part of EMR cluster, then ONLY install on the MASTER node

if [ $IS_MASTER_OR_NON_EMR -eq 1 ]
then
    sudo mkdir /mnt/YOUR_MOUNT_POINT_DIR

    sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-YOUR_EFS_ID.efs.us-east-1.amazonaws.com:/ /mnt/YOUR_MOUNT_POINT_DIR

    . /mnt/YOUR_MOUNT_POINT_DIR/bin/installGui.sh
fi

exit 0