Pablo Iranzo Gómez's blog

ago 28, 2015

Filtering email with imapfilter

Since some time ago, email filter management was not scaling for me as I was using server-side filtering, I had to deal with the web-based interface which was missing some elements like drag&drop reordering of rules, cloning, etc.

As I was already using offlineimap to sync from the remote mailserver to my system into a maildir folder, I had almost all the elements I needed.

After searching for several options imapfilter seemed to be a perfect fit, so I started with a small set of rules and start integration with my email process.

On my first attempts, I setup a pre-sync hook on offlineimap by using as well as the postsync hook I already had:

presynchook  = time imapfilter
postsynchook = ~/.mutt/postsync-offlineimap.sh

Initial attempts were not good at all, applying filters on the remote imapserver was very time consuming and my actual 1 minute delay after finishing one check was becoming a real 10-15 minute interval between checks because of the imapfiltering and this was not scaling as I was putting new rules.

After some tries, and as I already had all the email synced offline, moved filtering to be locally instead of server-side, but as imapfilter requires an imap server, I tricked dovecot into using the local folder to be offered via imap:

protocols = imap
mail_location = maildir:~/.maildir/FOLDER/:INBOX=~/.maildir/FOLDER/.INBOX/
auth_debug_passwords=yes

This also required to change my foldernames to use "." in front of them, so I needed to change mutt configuration too for this:

set mask=".*"

and my mailfoders script:

set mbox_type=Maildir
set folder="~/.maildir/FOLDER"
set spoolfile="~/.maildir/FOLDER/.INBOX"

#mailboxes `echo -n "+ "; find ~/.cache/notmuch/mutt/results ~/.maildir/FOLDER -type d -not -name 'cur' -not -name 'new' -not -name 'tmp' -not -name '.notmuch' -not -name 'xapian' -not -name 'FOLDER' -printf "+'%f' "`

mailboxes `find ~/.maildir/FOLDER -type d -name cur -printf '%h '|tr " " "\n"|grep -v "^/home/iranzo/.maildir/FOLDER$"|sort|xargs echo`
#Store reply on current folder
folder-hook . 'set record="^"'

After this, I could start using imapfilter and start working on my set of rules... but first problem appeared, apparently I started having some duplicated email as I was cancelling and rerunning the script while debugging so a new tool was also introduced to 'dedup' my imap folder named IMAPdedup with a small script:

#!/bin/bash
(
for folder in $(python ~/.bin/imapdedup.py -s localhost  -u iranzo    -w '$PASSWORD'  -m -c -v  -l)
do
    python ~/.bin/imapdedup.py -s localhost  -u iranzo    -w '$PASSWORD'  -m -c  "$folder"

done
) 2>&1|grep "will be marked as deleted"

This script was taking care of listing all email foders on 'localhost' with my username and password (can be scripted or use external tools to gather it) and dedup email after each sync (in my postsync-offlinemap.sh as well as lbdq script for fetchning new addresses, notmuch and running imapfilter after syncing (to cath the limited filtering I do sever-side)

I still do some server-side filtering (4 rules), to get on a "Pending sort" folder all email which is either:

  • New support cases remain at INBOX
  • All emails from case updates, bugzilla, etc to _pending
  • All emails containing 'list' or 'bounces' in from to _pending
  • All emails not containing me directly on CC or To, to _pending

This more or less ensures a clean INBOX with most important things still there, and easier rule handling for email sorting.

So, after some tests, this is at the moment a simplified version of my filtering file:

---------------
--  Options  --
---------------

options.timeout = 30
options.subscribe = true
options.create = false

function offlineimap (key)
    local status
    local value
    status, value = pipe_from('grep -A2 ACCOUNT ~/.offlineimaprc | grep -v ^#|grep '.. key ..'|cut -d= -f2')C
        value = string.gsub(value, ' ', '')
        value = string.gsub(value, '\n', '')
        return value
end

----------------
--  Accounts  --
----------------

-- Connects to "imap1.mail.server", as user "user1" with "secret1" as
-- password.
EXAMPLE = IMAP {
    server = 'localhost',
    username = 'iranzo',
    password = '$PASSWORD',
    port = 143
}
-- My email
myuser = 'ranzo'

function mine(messages)
    email=messages:contain_cc(myuser)+messages:contain_to(myuser)+messages:contain_from(myuser)
    return email
end

function filter(messages,email,destination)
    messages:contain_from(email):move_messages(destination)
    messages:contain_to(email):move_messages(destination)
    messages:contain_cc(email):move_messages(destination)
    messages:contain_field('sender', email):move_messages(destination)
end

function deleteold(messages,days)
    todelete=messages:is_older(days)-mine(messages)
    todelete:move_messages(EXAMPLE['Trash'])
end


-- Define the msgs we're going to work on

-- Move sent messages to INBOX to later sorting
sent = EXAMPLE.Sent:select_all()
sent:move_messages(EXAMPLE['INBOX'])

inbox = EXAMPLE['INBOX']:select_all()
pending = EXAMPLE['INBOX/_pending']:select_all()
todos = pending + inbox

-- Mark as read messages sent from my user
todos:contain_from(myuser):is_recent():mark_seen()

-- Delete google calendar forwards
todos:contain_to('piranzo@gapps.example.com'):delete_messages()

-- Move all spam messages to Junk folder
spam = todos:contain_field('X-Spam-Score','*****')
spam:move_messages(EXAMPLE['Junk'])

-- Move Jive notifications
filter(todos,'jive-notify@example.com',EXAMPLE['INBOX/EXAMPLE/Customers/_jive'])

-- Filter EXAMPLEN
filter(todos,'dev-null@rhn.example.com',EXAMPLE['Trash'])

-- Filter PNT
filter(todos:contain_subject('[PNT] '),'noreply@example.com',EXAMPLE['Trash'])

-- Filter CPG (Customer Private Group)
filter(todos:contain_subject('Red Hat - Group '),'noreply@example.com',EXAMPLE['INBOX/EXAMPLE/Customers/Other/CPG'])

-- Remove month start reminders
todos:contain_subject('mailing list memberships reminder'):delete_messages()

-- Delete messages about New accounts created (RHN)
usercreated=todos:contain_subject('New Red Hat user account created')*todos:contain_from('noreply@example.com')
usercreated:delete_messages()

-- Search messages from CPG's
cpg = EXAMPLE['INBOX/EXAMPLE/Customers/Other/CPG']:select_all()
cpg:contain_subject('Cust1'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust1/CPG'])
cpg:contain_subject('Cust2'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust2/CPG'])
cpg:contain_subject('Cust3'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust3/CPG'])
cpg:contain_subject('Cust4'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust4/CPG'])

-- Move bugzilla messages
filter(todos:contain_subject('] New:'),'bugzilla@example.com',EXAMPLE['INBOX/EXAMPLE/Customers/_bugzilla/new'])
filter(todos,'bugzilla@example.com',EXAMPLE['INBOX/EXAMPLE/Customers/_bugzilla'])

-- Move all support messages to Other for later processing
filter(todos:contain_subject('(NEW) ('),'support@example.com',EXAMPLE['INBOX/EXAMPLE/Customers/_new'])
filter(todos:contain_subject('Case '),'support@example.com',EXAMPLE['INBOX/EXAMPLE/Customers/Other/cases'])

EXAMPLE['INBOX/EXAMPLE/Customers/_new']:is_seen():move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Other/cases'])

support = EXAMPLE['INBOX/EXAMPLE/Customers/Other/cases']:select_all()
-- Restart the search only for messages in Other to also process if we have new rules

support:contain_subject('is about to breach its SLA'):delete_messages()
support:contain_subject('has breached its SLA'):delete_messages()
support:contain_subject(' has had no activity in '):delete_messages()

-- Here the process is customer after customer and mark as read messages from non-prio customers
support:contain_body('Cust1'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust1/cases'])
support:contain_body('Cust2'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust2/cases'])
support:contain_body('Cust3'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust3/cases'])
support:contain_body('Cust4'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust4/cases'])

-- For customer swith common matching names, use header field
support:contain_field('X-SFDC-X-Account-Number', 'XXXX'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust5/cases'])
support:contain_body('Customer         : COMMONNAME'):move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust6/cases'])

-- Non prio customers (mark updates as read)
cust7 = support:contain_body('WATCHINGCUST') + support:contain_body('Cust7')
cust7:mark_seen()
cust7:move_messages(EXAMPLE['INBOX/EXAMPLE/Customers/Cust7/cases'])

-- Filter other messages by domain
filter(todos,'todos.es', EXAMPLE['INBOX/EXAMPLE/Customers/Cust8'])

-- Process all remaining messages in INBOX + all read messages in pending-sort for mailing lists and move to lists folder
filter(todos,'list', EXAMPLE['INBOX/Lists'])
filter(todos,'bounces',EXAMPLE['INBOX/Lists'])

-- Add EXAMPLE lists, inbox and _pending and Fedora default bin for reprocessing in case a new list has been added
lists = todos + EXAMPLE['INBOX/Lists']:select_all() + EXAMPLE['INBOX/Lists/Fedora']:select_all()

-- Mailing lists

-- EXAMPLE
filter(lists,'outages-list',EXAMPLE['INBOX/Lists/EXAMPLE/general/outage'])
filter(lists,'announce-list',EXAMPLE['INBOX/Lists/EXAMPLE/general/announce'])

-- Fedora
filter(lists,'kickstart-list',EXAMPLE['INBOX/Lists/Fedora/kickstart'])
filter(lists,'ambassadors@lists.fedoraproject.org',EXAMPLE['INBOX/Lists/Fedora/Ambassador'])
filter(lists,'infrastructure@lists.fedoraproject.org',EXAMPLE['INBOX/Lists/Fedora/infra'])
filter(lists,'announce@lists.fedoraproject.org',EXAMPLE['INBOX/Lists/Fedora/announce'])
filter(lists,'lists.fedoraproject.org',EXAMPLE['INBOX/Lists/Fedora'])

-- OSP
filter(lists,'openstack@lists.openstack.org',EXAMPLE['INBOX/Lists/OpenStack'])
filter(lists,'openstack-es@lists.openstack.org',EXAMPLE['INBOX/Lists/OpenStack/es'])

-- Filter my messages not filtered back to INBOX
mios=pending:contain_from(myuser)
mios:move_messages(EXAMPLE['INBOX'])

-- move messages we're in BCC to INBOX for manual sorting
hidden = pending - mine(pending)
hidden:move_messages(EXAMPLE['INBOX'])

-- Start processing of messages older than:
maxage=60

-- Delete old messages from mailing lists
deleteold(EXAMPLE['INBOX/Lists/EXAMPLE/general/media'],maxage)
deleteold(EXAMPLE['INBOX/Lists/EXAMPLE/general/outage'],maxage)

-- delete old cases
maxage=180

-- for each in $(cat .imapfilter/config.lua|grep -i cases|tr " ,()" "\n"|grep cases|sort|uniq|grep -v ":" );do echo "deleteold($each,maxage)";done
deleteold(EXAMPLE['INBOX/EXAMPLE/Customers/Cust1/cases'],maxage)
deleteold(EXAMPLE['INBOX/EXAMPLE/Customers/Cust2/cases'],maxage)
deleteold(EXAMPLE['INBOX/EXAMPLE/Customers/Cust3/cases'],maxage)
deleteold(EXAMPLE['INBOX/EXAMPLE/Customers/Other/cases'],maxage)

deleteold(EXAMPLE['INBOX/EXAMPLE/Customers/_bugzilla'],maxage)

-- Empty trash every 7 days
maxage=7
deleteold(EXAMPLE['Trash'],maxage)

As this is applied filtering twice, offlineimap might be uploading part of your changes already, making it faster to next syncs, and suffle some of your emails while it runs.

The point of adding the already filtered set to be filtered again (CPG, cases, etc) is that if a new customer is consiredered to be filter on a folder of its own, the messages will be picked up and moved accordingly automatically ;-)

Hope it helps, and happy filtering!

Click to read and post comments

jul 17, 2015

RHEV-M with nested VM for OSP

Since some time ago, I've been mostly dealing with OpenStack, requiring different releases to test for different tests, etc.

Virtualization, as provided by KVM requires some CPU flags to get accelerated operations, vmx and svm depending on your processor architecture, but, of course, this is only provided on bare-metal.

In order to get more flexibility at the expense of performance, nestedvt allows to expose those flags to the VM's running at the hypervisor so you can run another level of VM's inside those VM's (this starts to sound like the movie Inception).

The problem, so far is that this required changes on the kernel and drivers to make it work, and was lacking lot of stability, so this is something NOT SUPPORTED FOR PRODUCTION USE but which makes perfect sense for demo environments, labs, etc, allowing you to maximize the use of your hardware for better flexibility but at the cost of performance.

As I was using RHEV for managing my home-lab I hit the first issue, my hypervisors (HP Proliant G7 N54L) where using RHEL-6 as operating system, and the support for nested was not very good, but luckily, RHEV-M 3.5 includes support for hypervisors running on RHEL-7, enabling to use latest features included in kernel, networking stack, etc.

First step, was to redeploy the servers, wasn't that hard, but required some extra steps as I had another unsupported approach (servers were sharing local storage over NFS for providing Storage Domains to environment, HIGHLY UNSUPPORTED), so I moved them from NFS to iSCSI provided by an external server and with the help of the kickstart I use for other systems, I started the process.

Once the two servers were migrated, the last one, finished moving VM's from NFS to iSCSI and needed to be put on maintenance and enable the other two (as a safety measure, RHEL-6 and RHEL-7 hosts cannot coexist on the same cluster in RHEV).

From here, just needed to enable NestedVT on the environment.

NestedVT 'just' requires to expose the svm or vmx flag to the VM running directly from the bare-metal host, and we need to do that for every VM we start. On normal system with libvirt, we can just edit the XML for the VM definition and define the CPU like this:

<cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G3</model>
    <feature policy='require' name='svm'/>
</cpu>

For RHEV, however, we don't have an XML we can edit, as it is created dynamically with the contents of the database for the VM (disks, NICS, name, etc), but we've the VDSM-Hooks mechanism for doing this.

Hooks in vdsm are a powerful and dangerous tool, as they can modify in-flight the XML used to create the VM, and allow lot of features to be implemented.

In the past, for example, those hooks could be used to provide DirectLUN support to RHEV, or fixed BIOS Serial Number for VM's where the product was still lacking the official feature, and in this case, we'll use them to provide the CPU flags we need.

As you can imagine, this is something that has lot of interested people behind, and we can find upstream a repository with VDSM-Hooks.

In this case, the one that we're needing is 'nestedvt', so we can proceed to install it on our hosts like:

wget http://mirrors.ibiblio.org/ovirt/pub/ovirt-3.4/rpm/el7/noarch/vdsm-hook-nestedvt-4.14.17-0.el7.noarch.rpm
rpm -Uvh vdsm-hook-nestedvt-4.14.17-0.el7.noarch.rpm

You'll need to put a host in maintenance and activate for VDSM to refresh the hooks installed and start new VM so we have the hook injecting the XML.

After it boots, egrep 'svm|vmx' /proc/pcuinfo should show the flags there.

But wait...

RHEV also includes a security feature that makes it impossible for a VM to spy on the communications meant to other VM's that makes it impossible to simulate other MAC's within it, and this is performed via libvirt filters on the interfaces.

To come to our rescue, another hook comes to play in, this time macspoof which allows to disable this security measure for a VM so it can execute virtualization within.

First, let's repeat the procedure and install the hook on all of our hypervisors:

wget http://mirrors.ibiblio.org/ovirt/pub/ovirt-3.4/rpm/el7/noarch/vdsm-hook-macspoof-4.14.17-0.el7.noarch.rpm
rpm -Uvh vdsm-hook-macspoof-4.14.17-0.el7.noarch.rpm

This will enable the hook in the system, but we also need to make the RHEV-M Engine aware of it, so we need to define a new Custom Property for VM's:

engine-config -s "UserDefinedVMProperties=macspoof=(true|false)"

This will ask us for the compatibility version (we'll choose 3.5) and enable a new true/false property for VM's that require this security measure lifted. We're doing of course this approach instead of disabling it for everyone to limit it's use to just the VM's needing it, not losing all the benefits on security provided.

As a side note, macspoof plugin is available in official repositories for RHEL7 hypervisor, so you can use this instead of oVirt's repo one.

Now when we create a new VM, for example to use with OpenStack, we can go to custom properties for this vm, select 'macspoof' and set a value of 'true' and once the VM is started will be able to see the processor extensions for virtualization and at the same time, the VM's created within, will be able to communicate with the outside world.

Enjoy!

Click to read and post comments

jun 26, 2015

Writing a Telegram.org bot in Python

Hi,

Telegram.org recently announced the support for writing bots for their platform, by providing details at https://core.telegram.org/bots.

I was missing for a long time the ability to get a count on karma like we've on irc servers, so I started with it.

My first try is published at github repo in https://github.com/iranzo/stampython.

At the moment it just uses the polling inteface to check the new messages received on the channels the bot is in, and later processes them and send the relevant replies via messages.

Also, some other commands are missing like the ones on redken that we use on IRC, but at least, basic functionality is there and is usable.

Enjoy!

Pablo

BTW: the bot is not allowed to join channels (@stampy_bot) so it remains in a controlled environment until the code is made more robust, but I'm thinking about having a second public instance on Openshift.redhat.com for wider audience.

Click to read and post comments

may 01, 2015

Intel AMT on Linux for remote control/fencing

Hi,

Some time ago, and after discussing with a colleague, I had a look on Intel's AMT, and this week I demoed it for another colleague as a cheap-replacement for having power fencing capabilities on commodity hardware.

AMT provides a server-like Out of band management like iLO, iDrac, RSB etc and it's included in i3 with vPro processors/chipsets of some equipment.

I did the test on a Lenovo X200/201 system I had as old laptop.

The steps used for configuring it, require to:

  • first enable the support in the BIOS, usually named 'Intel AMT' or 'Intel Active Management Technology'.
  • After this step it was possible to use the command to enter the special AMT firmware Intel(R) Management Engine which on this laptop is enabled with CTRL-P.
  • If this is the first time you enable it, you'll require to change the default admin password to something secure, usually mixed upper-lower case, symbol and numbers.
    • For this example we'll be using Qwer123$ as password.
  • Explore the settings, enable it and validate network settings.
    • I've enabled DHCP on both LAN and Wireless for IPv4 and IPv6, and enabled KVM redirection
  • Once finished, save changes and exit from firmware screen and let the system boot.

From another host, you can perform the remaining configuration steps, from now on, the 'target' system will be intercepting packets sent to specific port via the network cards and redirect to AMT firmware instead of going to target host. This is something important to note, the packets are only intercepting when coming from OUTSIDE the host so we'll use a second computer to access it.

You can use a browser pointing to target system's IP at port 16992, for example: http://target:16992

From that web interface and once logging with admin and the password set Qwer123$ we can continue doing some configuration, like the power states to control (for example, this laptop could be remotely powered when it was with the charger connected even if laptop was powered off).

Now, for doing the 'command-line' part, we will need to install one package on our system and rum some scripts.

# First we'll install amtterm wsmancli

dnf -y install amtterm wsmancli

# This will provide the two commands we'll later use, wsman for configuration and amttool for power control

# We need to define the host to use and password as well as the password we'll use for console redirection (via VNC)

AMT_PASSWORD='Qwer123$'
AMT_HOST=target
VNC_PASSWORD='Qwer123$'

# we can define those vars (specially AMT_PASSWORD) in our .profile or .bash_profile in order to avoid typing them everytime

# set the vnc password (must be 8 characters MAX)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k RFBPassword=${VNC_PASSWORD}

# enable KVM redirection to port 5900 (this will also intercept 5900 port for console redirection, so make it sure you'll not need it later)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k Is5900PortEnabled=true

# disable opt-in policy (do not ask user for console access)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k OptInPolicy=false

# disable session timeout (do not timeout sessions)
wsman put http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k SessionTimeout=0

# enable KVM (enable keyboard/video/monitor redirection)
wsman invoke -a RequestStateChange http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD} -k RequestedState=2

# OPTIONAL: view settings (validate all the settings)
wsman get http://intel.com/wbem/wscim/1/ips-schema/1/IPS_KVMRedirectionSettingData -h ${AMT_HOST} -P 16992 -u admin -p ${AMT_PASSWORD}

After this step, we should be able to use vinagre target to access the KVM redirection and remotely control our system.

For example, to control power of host you can use:

# Check host status:
amttool $AMT_HOST info

# Power up a powered-off host:
amttool $AMT_HOST powerup

# Power down a powered-on host:
amttool $AMT_HOST powerdown

Check man amttool for other commands like reset, powercycle.

IMPORTANT: note that some power state changes can only be performed based on previous status, you can check with info the available ones and current status of system.

As a bonus, there's a RFE1 for requesting this tool to be incorporated as power fencing mechanism in fence-agents once 'amtterm' is included in RHEL, in the meantime it's already available in Fedora, and when it comes to RHEL, hopefully could also be used as fence agent for Clusters and RHEV.

Enjoy!


  1. Request for Enhancement: a bugzilla request oriented not to fix a bug, but to incorporate new functionality/software into a product. 

Click to read and post comments

abr 01, 2015

Migrate SPIP-RSS post feed to HTML

I had my old blog based on SPIP, and I wanted to keep all the posts together, to make it easier to migrate in the future.

Initially, I migrated my posts from blogger, where there's an option to export the contents and some plugins to allow easier importing to markdown files (to be used by Octopress), those were the recent posts, so part of the job was already done there.

Next step, was to migrate old posts on my spip site.

SPIP, being not as popular as other solutions, might lack plugins for importing the data, but has a nice feature: it allows to provide full article contents via RSS.

So:

  • I entered into my site private area /ecrire/
  • Entered to the administration section and under Content, I temporarly changed the syndication settings to provide full articles instead of just summary.
  • Then, I visited the url for my user, but on the rss generator template: spip.php?page=backend&id_rubrique=6, and saved it as file.xml

At this point I needed some software for automating the initial conversion, so I went to python's feedparser libraries to perform this with a bit of coding:

url="/path/to/your/xml/file.xml"

import codecs
import feedparser
feed=feedparser.parse(url)

for item in feed["items"]:
    filename=item["date"][0:10]+"-"+item["link"][23:] #remove the first 23 chars from article url http+domain
    print filename
    with codecs.open(filename,'w','utf-8') as f:
        f.write("---\n")
        f.write("layout: post\n")
        for elem in ["title","date"]:
            f.write("%s: %s\n" % (elem,item[elem]))
        f.write("---\n")
        f.write(item["content"][0].value)

After each iteration, a new file was created using the old http link to the article (which already had stripped problematic characters).

Just moving those files to source/_posts allows me to republish them on a different site, and later work the conversion to markdown by using pandoc and some manual tuning.

Click to read and post comments
Next → Page 1 of 11