SLUUG Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
[[:home|Home]] | [[:projects|Projects]] | [[:hardwarerefresh_dec11|SLUUG Hardware Refresh]]

===== OpenNebula on Xen on OpenSuSE 12.1 =====

This is a working log - still in early draft. I'm still finalizing a lot of the config. If the formatting is confusing please disregard, I will be updating this as I make progress.

=== Create a VM on Xenpri1 ===
Create VM on xenpri -
virt-install -n nattylight -r 1024 --vcpu 1 --description "OpenNebula front-end, OpenSuSE 12.1 based" -l "http://download.opensuse.org/distribution/openSUSE-current/repo/oss/" --os-type linux --disk /etc/xen/images/nattylight.xvda,size=8 -w bridge=br0 --virt-type xen --arch x86_64 

-x brokenmodules=edd xencons=tty

using Yast, or vm-install -

name:nattyice
min-mem: 768
max-mem: 1024
vcpu: 2
description: OpenNebula front-end, openSuSE 12.1 based
Graphics: none
Virtual Disks: y
Disk type: 2 (hard disk)
Resides: /var/lib/xen/images/nattyice/xvda
size: 8g
Network adapters: add two
Bootable media: 1 (network URL)
Install URL: http://download.opensuse.org/distribution/openSUSE-current/repo/oss/

Enter through the rest

===YaST2===
Defaults unless otherwise stated
Timezone: chicago
Destkop: Other - Minimal Server Selection (Text Mode)
Suggest Partitioning: uncheck Propose separate home, check Use btrfs
Users: open-nebula-admin
Password: ON
  *Do not use for sys admin, no auto login
Root PW: NI
Enable Boot from MBR
Enable SSH
Software: install Patterns-openSUSE-webyast-{ui,ws}

Upon reboot - change hostname to nattyice



==== OpenNebula 3.1.0 ====
  *Get the OpenSuSE x86_64 tarball

<code>wget http://dev.opennebula.org/attachments/download/499/opennebula-3.1.0.tar.gz </code>

  *Install the packman repo
<code>$ zypper ar ftp://anorien.csc.warwick.ac.uk/packman/suse/openSUSE_12.1/ Packman</code>
<sub>Feel free to use a different mirror - http://packman.links2linux.de/mirrors </sub>

  * Add OpenNebula:Testing repo
<code> zypper ar http://download.opensuse.org/repositories/Virtualization:/Cloud:/OpenNebula:/Testing/openSUSE_12.1/ OpenNebula-Testing</code>

  * Add OpenNebula (may need to do this after install...)
<code>http://download.opensuse.org/repositories/Virtualization:/Cloud:/OpenNebula/openSUSE_12.1/</code>

**Install dependencies**
<code> zypper install gcc gcc-c++ make patch libopenssl-devel libcurl-devel scons pkg-config sqlite3-devel libxslt-devel libxmlrpc_server_abyss++3 libxmlrpc_client++3 libexpat-devel libxmlrpc_server++3 libxml2-devel ruby ruby-doc-ri ruby-doc-html ruby-devel rubygems xmlrpc-c libmysqlclient-devel rubygem-rake libxmlrpc++3</code>

**Install via openSuSE repos**
  * Add OpenNebula:Testing repo
<code> zypper ar http://download.opensuse.org/repositories/Virtualization:/Cloud:/OpenNebula:/Testing/openSUSE_12.1/ OpenNebula-Testing</code>

  * Add OpenNebula (may need to do this after install...)
<code>http://download.opensuse.org/repositories/Virtualization:/Cloud:/OpenNebula/openSUSE_12.1/</code>

  * Install dependencies
<code> zypper install gcc gcc-c++ make patch libopenssl-devel libcurl-devel scons pkg-config sqlite3-devel libxslt-devel libxmlrpc_server_abyss++3 libxmlrpc_client++3 libexpat-devel libxmlrpc_server++3 libxml2-devel ruby ruby-doc-ri ruby-doc-html ruby-devel rubygems xmlrpc-c libmysqlclient-devel rubygem-rake libxmlrpc++3 mysql-community-server</code>

==== MySQL install on openSuSE 12.1 ====

  *Install packages
<code>zypper install mysql-community-server</code>

  *Start service
<code>service mysql start</code>

  *Secure SQL 
<code>mysql_secure_installation</code>

  *Set root PW to the same as the root user PW, Select Y (defaults) for the rest

  *Create users with privileges 
<code>
$ mysql -u root -p
Enter password: 
Welcome to the MySQL monitor. [...]

mysql> GRANT ALL PRIVILEGES ON opennebula.* TO 'oneadmin' IDENTIFIED BY 'oneadmin';
Query OK, 0 rows affected (0.00 sec)
</code>

==== Back to Open Nebula ====

  * openSuSE apparently forgets the install_gems script from OpenNebula - you'll need to manually install the following gems
<code>gem install libxml-ruby json nokogiri rake xmlparser sqlite3 mysql curb thin uuid sinatra sequel amazon-ec2 </code>

  * Install OpenNebula and Web UI (sunstone)
<code>zypper install opennebula opennebula-sunstone</code>
<sub>Install dependencies as needed - ignore the rubygem-* files we will install these later</sub>

  * Take note of onedmin user ID and cloud groupid
<code> id oneadmin </code>
<sub>In This case it's UID=1000 GID=1000</sub>
  * Logon to the xen host add a user of with the same UID/GID and a home of /var/lib/one
<code>useradd --uid 100 -g cloud -d /var/lib/one oneadmin</code>
<sub>If there is a conflict use usermod --uid UID to change the user who is in conflict</sub>
  * Set user's password to the same as the front-end

User's home dirs will eventually be moved to the NFS share for now copy the keys to the home dir of the xen box

  * Create ssh keypair for oneadmin, use defaults - no passphrase
<code>ssh-keygen</code>

  * Add key to auth users
<code>$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code>

  * Change permissions of keys for to prevent issues later
<code>
$ chmod 700 ~/.ssh/
$ chmod 600 ~/.ssh/id_dsa.pub
$ chmod 600 ~/.ssh/id_dsa
$ chmod 600 ~/.ssh/authorized_keys
</code>

  * Create user ssh client config to prevent prompting for known hosts
<code>vim ~/.ssh/config

Host *
    StrictHostKeyChecking no

</code>

  * Set OpenNebula to use the same credentials as oneadmin user
<code>
$ mkdir ~/.one
$ echo "oneadmin:password" > ~/.one/one_auth
$ chmod 600 ~/.one/one_auth
</code>	

  * Start OneNebula == *Be sure to start as oneadmin user* ==
<code>one start</code>

  * Verify the oned is running and working
<code>onevm list</code>

==== Basic config ====

== Enable MySQL ==
  * Config /etc/one/oned.conf (you can just hash the example)
<code>

DB = [ backend = "mysql",
       server  = "localhost",
       port    = 0,
       user    = "oneadmin",
       passwd  = "oneadmin",
       db_name = "opennebula" ]
</code>

== Xen Host ==

  * Edit /etc/sudoers to allow oneadmin to control vms
<code>visudo

oneadmin    ALL=(ALL) NOPASSWD: /usr/sbin/xm *
oneadmin    ALL=(ALL) NOPASSWD: /usr/sbin/xentop *
oneadmin    ALL=(ALL) NOPASSWD: /usr/sbin/ovs_vsctl *
</code>

  * Edit /etc/one/oned.conf to use Xen Driver (hash the kvm config example and unhash the xen example)
<code>
    IM_MAD = [
        name       = "im_xen",
        executable = "one_im_ssh",
        arguments  = "xen" ]

    VM_MAD = [
        name       = "vmm_xen",
        executable = "one_vmm_exec",
        arguments  = "xen",
        default    = "vmm_exec/vmm_exec_xen.conf",
        type       = "xen" ]
</code>

  * install ruby on xen host (>=1.8.7)
<code>zypper install ruby</code>

== Storage ==

  * Map /etc/xen/images to btrfs raid 10 local array
  * Map /var/lib/one/ to NFS server 600G S/W raid 10 
  * Map /home/* to NFS server 600G S/W raid 10

  *On NFS Server be sure to perform these fixes since root user access will be needed

-Set the permissions in the directory and the images exported to very permissive (directories: a+rx, files: a+rw)
-Disable root squashing adding no_root_squash to nfs exporting options\\

/etc/exports/\\
<file>
/srv/storage/@/home     192.168.118.0/27(crossmnt,rw,no_root_squash,sync,no_subtree_check) 192.168.115.0/27(crossmnt,rw,no_root_squash,sync,no_subtree_check)
/srv/storage/@/backups  192.168.115.0/27(crossmnt,rw,no_root_squash,sync,no_subtree_check) 192.168.118.0/27(crossmnt,rw,no_root_squash,sync,no_subtree_check)
/var/lib/one    192.168.118.0/27(crossmnt,rw,no_root_squash,sync,no_subtree_check) 192.168.115.0/27(crossmnt,rw,no_root_squash,sync,no_subtree_check)
</file>

/etc/fstab (client side)
<file>
192.168.115.5:/var/lib/one /var/lib/one    nfs     rw,hard,intr
</file>

  * Recursive bind mount images data store. I for some reason am not getting crossmnt option to work on NFS

<code>
# mount --rbind /srv/storage/\@/images/ /var/lib/one/datastores/
</code>

/etc/fstab *** This Doesn't work either, still experimenting... Probably need to mount device directly since its a subvolume ***\\
<file>
/srv/storage/\@/images/ /var/lib/one/datastores/ none defaults,rbind
</file>

  * Edit /etc/one/oned.conf (default config)
<code>
TM_MAD = [
    name       = "tm_shared",
    executable = "one_tm",
    arguments  = "tm_shared/tm_shared.conf" ]
</code>

== Networking - Open vSwitch == 

  * Edit /etc/one/oned.conf (unhash open vSwitch VM_HOOK)
<code>
VM_HOOK = [
    name      = "openvswitch-vlan",
    on        = "RUNNING",
    command   = "vnm/openvswitch-vlan",
    arguments = "$TEMPLATE",
    remote    = "yes" ]
</code>


===**** INSTALL ON HOST(S) ****===
  * Install Open vSwitch from source since it has failed openSuSE build project
  * Install dependencies
<code> zypper install python-qt4 python-Twisted python-zope.interface tcpdump kernel-xen-base kernel-xen-devel autoconf automake dotconf gcc glibc libopenssl-devel make</code>

<sub>Packages for Debian: python-qt4 python-twisted python-zopeinterface tcpdump linux-headers autoconf automake libdotconf-dev libdotconf1.0 gcc glibc-source libc-dev libssl-dev

<sub>Switch to build user</sub>
  * Get tarball
<code>wget http://openvswitch.org/releases/openvswitch-1.2.2.tar.gz</code>

  * Untar, change to dir
<code>tar -xzf openvswitch-1.2.2.tar.gz
	cd openvswitch...</code>

  * Configure with unique options (at this time openvswitch will not compile kernel modules for kernel 3.1)
<code>./configure --prefix=/usr --localstatedir=/var </code>

  * Make, then su make install
<code>$make
#make install
</code>

  * Intialize configuration database
<code>#ovsdb-tool create /usr/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema</code>

  * Initialize configuration database server
<code>
ovsdb-server --remote=punix:/var/run/openvswitch/db.sock \
                     --remote=db:Open_vSwitch,manager_options \
                     --private-key=db:SSL,private_key \
                     --certificate=db:SSL,certificate \
                     --bootstrap-ca-cert=db:SSL,ca_cert \
                     --pidfile --detach
</code>

  * initialize database 
<code> ovs-vsctl ---no-wait init</code>

  * Start the Open vSwitch daemon
<code>ovs-vswitchd --pidfile --detach</code>

<sub>Note there are a few errors that we should be able to safely diregard since we did not compile the kernel module</sub>

***** need kernel modules for brctl layer

**vSwitch doesn't like kernel 3.1 - we'll likely need to use ebtables which will conflict with xenpri0**

=== Add a new host ===
It is critical that your opennebula front end can resolve the name of your hypervisors. In this scenario we will use the /etc/hosts file to do this for us.\\ 

<code># vim /etc/hosts

127.0.0.1 xenhost

:wq
</code>


Switch over to the oneadmin user and add your hosts.\\
To add a host you use the onehost create command. This command needs to know the information manager (im) driver, the virtual machine monitor (vmm) driver and the network driver that the host is using. In our case we will be adding a xen hypervisor, that is set up to use open vSwitch for its network driver.

<code># su - oneadmin
~$ onehost create xenhost -i im_xen -v vmm_xen -n ovswitch
</code>

Once complete you should now see your host on the onehost list, you can also use onehost show <hostid> to show the details of your host:\\

<code>
oneadmin@xenhost:~$ onehost list
  ID NAME         CLUSTER     RVM   TCPU   FCPU   ACPU   TMEM   FMEM   AMEM STAT
   1 xenhost     -             0      0      0    100     0K     0K     0K   on

oneadmin@xenhost:~$ onehost show xenhost
HOST 1 INFORMATION                                                              
ID                    : 1                   
NAME                  : xenhost            
CLUSTER               : -                   
STATE                 : MONITORED           
IM_MAD                : im_xen              
VM_MAD                : vmm_xen             
VN_MAD                : ovswitch            
LAST MONITORING TIME  : 1339032919          

HOST SHARES                                                                     
MAX MEM               : 16678912            
USED MEM (REAL)       : 13844480            
USED MEM (ALLOCATED)  : 0                   
MAX CPU               : 800                 
USED CPU (REAL)       : 107                 
USED CPU (ALLOCATED)  : 0                   
MAX DISK              : 0                   
USED DISK (REAL)      : 0                   
USED DISK (ALLOCATED) : 0                   
RUNNING VMS           : 0                   

MONITORING INFORMATION                                                          
ARCH="x86_64"
CPUSPEED="3100"
FREECPU="693"
FREEMEMORY="2834432"
HOSTNAME="xenhost"
HYPERVISOR="xen"
MODELNAME="AMD FX(tm)-8120 Eight-Core Processor "
NETRX="0"
NETTX="0"
TOTALCPU="800"
TOTALMEMORY="16678912"
USEDCPU="107"
USEDMEMORY="13844480"

</code>

====Work In Progress - 21DEC11====

[[:home|Home]] | [[:projects|Projects]] | [[:hardwarerefresh_dec11|SLUUG Hardware Refresh]]
SLUUG Wiki

User Tools

Site Tools

Page Tools
