Pablo Iranzo Gómez's blog

jul 17, 2015

RHEV-M with nested VM for OSP

Since some time ago, I've been mostly dealing with OpenStack, requiring different releases to test for different tests, etc.

Virtualization, as provided by KVM requires some CPU flags to get accelerated operations, vmx and svm depending on your processor architecture, but, of course, this is only provided on bare-metal.

In order to get more flexibility at the expense of performance, nestedvt allows to expose those flags to the VM's running at the hypervisor so you can run another level of VM's inside those VM's (this starts to sound like the movie Inception).

The problem, so far is that this required changes on the kernel and drivers to make it work, and was lacking lot of stability, so this is something NOT SUPPORTED FOR PRODUCTION USE but which makes perfect sense for demo environments, labs, etc, allowing you to maximize the use of your hardware for better flexibility but at the cost of performance.

As I was using RHEV for managing my home-lab I hit the first issue, my hypervisors (HP Proliant G7 N54L) where using RHEL-6 as operating system, and the support for nested was not very good, but luckily, RHEV-M 3.5 includes support for hypervisors running on RHEL-7, enabling to use latest features included in kernel, networking stack, etc.

First step, was to redeploy the servers, wasn't that hard, but required some extra steps as I had another unsupported approach (servers were sharing local storage over NFS for providing Storage Domains to environment, HIGHLY UNSUPPORTED), so I moved them from NFS to iSCSI provided by an external server and with the help of the kickstart I use for other systems, I started the process.

Once the two servers were migrated, the last one, finished moving VM's from NFS to iSCSI and needed to be put on maintenance and enable the other two (as a safety measure, RHEL-6 and RHEL-7 hosts cannot coexist on the same cluster in RHEV).

From here, just needed to enable NestedVT on the environment.

NestedVT 'just' requires to expose the svm or vmx flag to the VM running directly from the bare-metal host, and we need to do that for every VM we start. On normal system with libvirt, we can just edit the XML for the VM definition and define the CPU like this:

<cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G3</model>
    <feature policy='require' name='svm'/>
</cpu>

For RHEV, however, we don't have an XML we can edit, as it is created dynamically with the contents of the database for the VM (disks, NICS, name, etc), but we've the VDSM-Hooks mechanism for doing this.

Hooks in vdsm are a powerful and dangerous tool, as they can modify in-flight the XML used to create the VM, and allow lot of features to be implemented.

In the past, for example, those hooks could be used to provide DirectLUN support to RHEV, or fixed BIOS Serial Number for VM's where the product was still lacking the official feature, and in this case, we'll use them to provide the CPU flags we need.

As you can imagine, this is something that has lot of interested people behind, and we can find upstream a repository with VDSM-Hooks.

In this case, the one that we're needing is 'nestedvt', so we can proceed to install it on our hosts like:

wget http://mirrors.ibiblio.org/ovirt/pub/ovirt-3.4/rpm/el7/noarch/vdsm-hook-nestedvt-4.14.17-0.el7.noarch.rpm
rpm -Uvh vdsm-hook-nestedvt-4.14.17-0.el7.noarch.rpm

You'll need to put a host in maintenance and activate for VDSM to refresh the hooks installed and start new VM so we have the hook injecting the XML.

After it boots, egrep 'svm|vmx' /proc/pcuinfo should show the flags there.

But wait...

RHEV also includes a security feature that makes it impossible for a VM to spy on the communications meant to other VM's that makes it impossible to simulate other MAC's within it, and this is performed via libvirt filters on the interfaces.

To come to our rescue, another hook comes to play in, this time macspoof which allows to disable this security measure for a VM so it can execute virtualization within.

First, let's repeat the procedure and install the hook on all of our hypervisors:

wget http://mirrors.ibiblio.org/ovirt/pub/ovirt-3.4/rpm/el7/noarch/vdsm-hook-macspoof-4.14.17-0.el7.noarch.rpm
rpm -Uvh vdsm-hook-macspoof-4.14.17-0.el7.noarch.rpm

This will enable the hook in the system, but we also need to make the RHEV-M Engine aware of it, so we need to define a new Custom Property for VM's:

engine-config -s "UserDefinedVMProperties=macspoof=(true|false)"

This will ask us for the compatibility version (we'll choose 3.5) and enable a new true/false property for VM's that require this security measure lifted. We're doing of course this approach instead of disabling it for everyone to limit it's use to just the VM's needing it, not losing all the benefits on security provided.

As a side note, macspoof plugin is available in official repositories for RHEL7 hypervisor, so you can use this instead of oVirt's repo one.

Now when we create a new VM, for example to use with OpenStack, we can go to custom properties for this vm, select 'macspoof' and set a value of 'true' and once the VM is started will be able to see the processor extensions for virtualization and at the same time, the VM's created within, will be able to communicate with the outside world.

Enjoy!

Click to read and post comments

jun 26, 2015

Writing a Telegram.org bot in Python

Hi,

Telegram.org recently announced the support for writing bots for their platform, by providing details at https://core.telegram.org/bots.

I was missing for a long time the ability to get a count on karma like we've on irc servers, so I started with it.

My first try is published at github repo in https://github.com/iranzo/stampython.

At the moment it just uses the polling inteface to check the new messages received on the channels the bot is in, and later processes them and send the relevant replies via messages.

Also, some other commands are missing like the ones on redken that we use on IRC, but at least, basic functionality is there and is usable.

Enjoy!

Pablo

BTW: the bot is not allowed to join channels (@stampy_bot) so it remains in a controlled environment until the code is made more robust, but I'm thinking about having a second public instance on Openshift.redhat.com for wider audience. You can invite the public instance by inviting @redken_bot

Click to read and post comments

may 01, 2015

Intel AMT on Linux for remote control/fencing

Hi,

Some time ago, and after discussing with a colleague, I had a look on Intel's AMT, and this week I demoed it for another colleague as a cheap-replacement for having power fencing capabilities on commodity hardware.

AMT provides a server-like Out of band management like iLO, iDrac, RSB etc and it's included in i3 with vPro processors/chipsets of some equipment.

I did the test on a Lenovo X200/201 system I had as old laptop.

The steps used for configuring it, require to:

  • first enable the support in the BIOS, usually named 'Intel AMT' or 'Intel Active Management Technology'.
  • After this step it was possible to use the command to enter the special AMT firmware Intel(R) Management Engine which on this laptop is enabled with CTRL-P.
  • If this is the first time you enable it, you'll require to change the default admin password to something secure, usually mixed upper-lower case, symbol and numbers.
    • For this example we'll be using Qwer123$ as password.
  • Explore the settings, enable it and validate network settings.
    • I've enabled DHCP on both LAN and Wireless for IPv4 and IPv6, and enabled KVM redirection
  • Once finished, save changes and exit from firmware screen and let the system boot.

From another host, you can perform the remaining configuration steps, from now on, the 'target' system will be intercepting packets sent to specific port via the network cards and redirect to AMT firmware instead of going to target host. This is something important to note, the packets are only intercepting when coming from OUTSIDE the host so we'll use a second computer to access it.

You can use a browser pointing to target system's IP at port 16992, for example: http://target:16992

From that web interface and once logging with admin and the password set Qwer123$ we can continue doing some configuration, like the power states to control (for example, this laptop could be remotely powered when it was with the charger connected even if laptop was powered off).

Now, for doing the 'command-line' part, we will need to install one package on our system and rum some scripts.

# First we'll install amtterm wsmancli

dnf -y install amtterm wsmancli

# This will provide the two commands we'll later use, wsman for configuration and amttool for power control

# We need to define the host to use and password as well as the password we'll use for console redirection (via VNC)

AMT_PASSWORD='Qwer123$'
AMT_HOST=target
VNC_PASSWORD='Qwer123$'

# we can define those vars (specially AMT_PASSWORD) in our .profile or .bash_profile in order to avoid typing them everytime

# set the vnc password (must be 8 characters MAX)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k RFBPassword=${VNC_PASSWORD}

# enable KVM redirection to port 5900 (this will also intercept 5900 port for console redirection, so make it sure you'll not need it later)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k Is5900PortEnabled=true

# disable opt-in policy (do not ask user for console access)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k OptInPolicy=false

# disable session timeout (do not timeout sessions)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k SessionTimeout=0

# enable KVM (enable keyboard/video/monitor redirection)
wsman invoke -a RequestStateChange http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k RequestedState=2

# OPTIONAL: view settings (validate all the settings)
wsman get http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD}

After this step, we should be able to use vinagre target to access the KVM redirection and remotely control our system.

For example, to control power of host you can use:

# Check host status:
amttool $AMT_HOST info

# Power up a powered-off host:
amttool $AMT_HOST powerup

# Power down a powered-on host:
amttool $AMT_HOST powerdown

Check man amttool for other commands like reset, powercycle.

IMPORTANT: note that some power state changes can only be performed based on previous status, you can check with info the available ones and current status of system.

As a bonus, there's a RFE1 for requesting this tool to be incorporated as power fencing mechanism in fence-agents once 'amtterm' is included in RHEL, in the meantime it's already available in Fedora, and when it comes to RHEL, hopefully could also be used as fence agent for Clusters and RHEV.

Enjoy!


  1. Request for Enhancement: a bugzilla request oriented not to fix a bug, but to incorporate new functionality/software into a product. 

Click to read and post comments

abr 01, 2015

Migrate SPIP-RSS post feed to HTML

I had my old blog based on SPIP, and I wanted to keep all the posts together, to make it easier to migrate in the future.

Initially, I migrated my posts from blogger, where there's an option to export the contents and some plugins to allow easier importing to markdown files (to be used by Octopress), those were the recent posts, so part of the job was already done there.

Next step, was to migrate old posts on my spip site.

SPIP, being not as popular as other solutions, might lack plugins for importing the data, but has a nice feature: it allows to provide full article contents via RSS.

So:

  • I entered into my site private area /ecrire/
  • Entered to the administration section and under Content, I temporarly changed the syndication settings to provide full articles instead of just summary.
  • Then, I visited the url for my user, but on the rss generator template: spip.php?page=backend&id_rubrique=6, and saved it as file.xml

At this point I needed some software for automating the initial conversion, so I went to python's feedparser libraries to perform this with a bit of coding:

url="/path/to/your/xml/file.xml"

import codecs
import feedparser
feed=feedparser.parse(url)

for item in feed["items"]:
    filename=item["date"][0:10]+"-"+item["link"][23:] #remove the first 23 chars from article url http+domain
    print filename
    with codecs.open(filename,'w','utf-8') as f:
        f.write("---\n")
        f.write("layout: post\n")
        for elem in ["title","date"]:
            f.write("%s: %s\n" % (elem,item[elem]))
        f.write("---\n")
        f.write(item["content"][0].value)

After each iteration, a new file was created using the old http link to the article (which already had stripped problematic characters).

Just moving those files to source/_posts allows me to republish them on a different site, and later work the conversion to markdown by using pandoc and some manual tuning.

Click to read and post comments

mar 28, 2015

Install RHEL7/Centos/Fedora on a software raid device

Installing Linux on a RAID has lot of advantages, from using RAID1 to enjoy protection against drive failures or RAID0 to combine the size of several drives to create bigger space for files with all the smaller disks we have.

There are several RAID level definitions and may have different uses depending on our needs and hardware availability.

For this, I focused on using raid1 for the system disks (for greater redundancy/protection against failures) and raid0 (for combining several disks to make bigger space available for non important data)..

Why or why not use a RAID via software

Pros

  • There's no propietary data on the disks that could require this specific controller in case the hardware fails.
  • Can be performed on any system, disk combination, etc

Cons

  • The use of dedicated HW RAID cards allows to offload the CPU intensive tasks for raid calculation, etc to the dedicated processor, freeing internal CPU for system/user usage.
  • Dedicated cards may have fancier features that require no support from the operating system as are all implemented by the card itself and presented to the OS as a standard drive.

Performing the setup

As I was installing on a HP Microserver G8 recently, I had to first disable the advanced mode for the included controller, so it behaved like a standard SATA one, once done, I was able to boot from my OS image (in this case EL7 iso).

Once the ISO is booted in rescue mode, I could switch to the second console with ALT-F2 so I could start executing commands on the shell.

First step is to setup partitioning, in this case I did two partitions, first one for holding /boot and the second one for setting up the LVM physical volume where the other Logical Volumes will be defined later.

I've elected this setup over others because mdadm allows transparent support for booting (grub supports booting form it) and easy to manage setup.

For partitions, remember to allocate at least 500mb for /boot and as much as needed for your SO, for example, if only base OS is expected to have RAID protection, having a 20Gb partition will be enough, leaving the remaining disk to be used for a RAID0 device for allocating non-critical files.

For both partitions, set type with fdisk to fd: Linux RAID autodetect, and setup the two drives we'll use for initial setup using the same values, for example:

fdisk /dev/sda
n # for new partition
p # for primary
<ENTER> # for first sector
+500M # for size
t # for type
fd # for Linux RAID autodetect
n # new partition
p # primary
<ENTER>
+20G #for size
t #for type
2 # for select 2nd partition
fd # for Linux RAID autodetect
# n for new partition
p # for primary
<ENTER> # for first sector
<ENTER> # for remaining disk
t # for type
3 # for third partition
fd # for Linux RAID Autodetect
w # for Writing changes

And repeat that for /dev/sdb

At this point, we'll have both sda and sdb with the same partitions defined: sd{a,b}1 with 500Mb for /boot and sd{a,b}2 with 20Gb for LVM and the remaining disk for RAID0 LVM.

Now, it's time to create the raid device on top, for simplicity, I tend to use md0 for /boot, so let's start with it.

Creating the raid devices with Multiple Devices mdadm

Let's create the raid devices for each system, starting with /boot:

mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda2 /dev/sdb2
mdadm --create /dev/md2 --level=0 --raid-devices=2 /dev/sda3 /dev/sdb3

Now, check the status of the raid device creation by issuing:

cat /proc/mdstat

Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda1[0] sdb1[1]
      534760 blocks level 1, 64k chunk, algorithm 2 [2/2] [UU]
            [==>..................]  recovery = 12.6% (37043392/292945152) finish=127.5min speed=33440K/sec
md1 : active raid1 sda2[0] sdb2[1]
      20534760 blocks level 1, 64k chunk, algorithm 2 [2/2] [UU]
            [=====>...............]  recovery = 25.9% (37043392/692945152) finish=627.5min speed=13440K/sec
...

When it finishes, all the devices will appear as synced, and we can start the installation of the operating system.

What I did, after this point, is to reboot the install media, so I could use anaconda installer to select manually the filesystems, creating /boot on /dev/md0, then the Physical Volume on /dev/md1 for the operating system.

Select the manual partitioning during the installation to define above devices as their intended usage, and once it has been installed, create the additional Physical volume on /dev/md2 and define the intended mountpoints, etc.

Enjoy!

Click to read and post comments
← Previous Next → Page 3 of 13