Pablo Iranzo Gómez's blog

jun 07, 2018

may 13, 2018

SuperSec 2018!

Presentation at SuperSec 2018!

I've got a slot at SuperSec 2018 (https://supersec.es/): Congress on secure software development, happening in Almería, Spain on the weekend on 12-13 May.

I'll be presenting on 13th may at 10:50, and the slide deck to be used is at https://github.com/citellusorg/citellus/blob/master/doc/supersec2018-presentation-ES.md and I'll be updating this once I get the recording URL.

Some data about the event:

About the topics, the main focus was security on software from design phase to production and maintenance.

Some of the presentations insisted on the costs not only for your brand, reputation or business damage, but also on the actual cost of fixing issues later on vs doing a secure development.

For the secure development approach, where most of the bugs are introduced and can be fixed for cheap, some tools were presented that help to early detect known coding mistakes, hilighting them in automated, and later, working on static code analysis and code review.

It was also interesting to see the EU focus via FOSSA-2 project to promote Open Source and hear about legal implications and intended roadmap for security in software, both for Spain and EU.

Common Criteria was also one of the topics as well as the code audits, penetration testing, etc (with over 20 slots you can imagine :) )

On our side, we were presenting about how Citellus can help in detecting current or future issues that affect your environment and how easily it can be extended to cover your use cases while contributing it back to community.

Of course we were also hilighting how collaboration got Citellus enhanced with feedback from RDO Project users to cover not only RHEL6 and RHEL7 but also Fedora, CentOS and other distributions via more generic tests and functios that covers them.

Enjoy!

PD: Hilighted in Citellus blog at https://citellus.org/blog/2018/04/16/supersec/

Click to read and post comments

ene 27, 2018

DevConf.cz 2018!

Presentation at Devconf.cz 2018!

As hilighted in the prior edition of the 'What's new', we got a slot for DevConf.cz 2018.

During that slot, my colleagues Martin, Pablo and myself were presenting on the history and basics of Citellus and how it helps on debugging issues and providing faster analysis of already known ones.

It is possible to watch the recording at https://www.youtube.com/watch?v=SDzzqrUdn5A and the slides used at: https://github.com/citellusorg/citellus/blob/master/doc/devconfcz2018-presentation-revealmd.md

Hope you enjoy it!

Click to read and post comments

ene 16, 2018

Recent changes in Magui and Citellus

What's new?

During recent weeks we've been coding and performing several changes to Citellus and Magui.

Checking the latest logs or list of issues open and closed on github is probably not an easy task or the best way to get 'up-to-date' with changes, so I'll try to compile a few here.

First of all, we're going to present it at Devconf.cz 2018, so come stop-by if assisting :-)

Some of the changes include...

Citellus

  • New functions for bash scripts!
    • We've created lot of functions to check different things:
      • installed rpm
      • rpm over specific version
      • compare dates over X days
      • regexp in file
      • etc..
    • Functions do allow to do quicker plugin development.
  • save/restore options so they can be loaded automatically for each execution
    • Think of enabled filters, excluded, etc
  • metadata added for plugins and returned as dictionary
  • plugin has a unique ID for all installations based on plugin relative path and plugin name
    • We do use that ID in magui to select the plugin data we'll be acting on
  • plugin priority!
    • Plugins are assigned a number between 0 and 1000 that represents how likely it's going to affect your environment, and you can filter also on it with --prio
  • extended via 'extensions' to provide support for other plugins
    • moved prior plugins to be core extension
    • ansible playbook support via ansible-playbook command
    • metadata plugins that just generate metadata (hostname, date for sosreport, etc)
  • Web Interface!!
    • David Valee Delisle did a great job on preparing an html that loads citellus.json and shows it graphically.
    • Thanks to his work, we did extended some other features like priority, categories, etc that are calculated via citellus and consumed via citellus-www.
    • Interface can also load magui.json (with ?json=magui.json) and show it's output.
    • We did extend citellus to take --web to automatically create the json named citellus.json on the folder specified with -o and copy the citellus.html file there. So if you provide sosreports over http, you can point to citellus.html to see graphical status! (check latest image at citellus website as www.png )
  • Increased plugin count!
    • Now we do have more than 119 across different categories
    • A new plugin in python reboot.py that checks for unexpected reboots
    • Spectre/Meltdown security checks!

Magui

  • If there's an existing citellus.json magui does load it to speed it up process across multiple sosreports.
  • Magui can also use ansible-playbook to copy citellus program to remote host and run there the command, and bring back the generated citellus.json so you can quickly run citellus across several hosts without having to manually perform operations or generate sosreports.
  • Moved prior data to two plugins:
    • citellus-outputs
      • Citellus plugins output arranged by plugin and sosreport
    • citellus-metadata
      • Outputs metadata gathered by metadata plugins in citellus arranged by plugin and sosreport
  • First plugins that compare data received from citellus on global level
    • Plugins are written in python and use each plugin id to just work on the data they know how to process
    • pipeline-yaml
      • Checks if pipeline.yaml and warns if is different across hosts
    • seqno
      • Checks latest galera seqno on hosts
    • release
      • Reports RHEL release across hosts and warns if is different across hosts
  • Enable quiet mode on the data received from citellus as well as local plugins, so only outputs with ERROR or different output on sosreports is shown, even on magui plugins.

Wrap up!

As you can see we've been busy trying to improve plugins, Citellus framework and Magui as well.

We've been also busy demonstrating to others it's value and raising lot of new issues and closing them with our commits (294 requests closed so far).

So, come and tell us what else are you missing or how can we improve it to suit your needs (or code them yourself and submit a review!)

Click to read and post comments

oct 26, 2017

i18n and 'bash8' in bash

Introduction

In order to improve Citellus and Magui, we did implement some Unit testing to improve code quality.

The tests written were made in python and with some changes it was also possible to validate the actual tests.

Also, we did prepare the strings in python using gettext library so the actual messages can be translated to the language of choice (defaults to en, but can be changed via --lang modifier of citellus).

Bashate for bash code validation

One of the things I did miss was to have some kind of tox8 for validate format, and locate some errors. After some research I came to bashate, and as it was written in python was very easy to integrate:

  • Update test-requirements.txt to request bashate for 'tests'
  • Editing tox.ini to add a new section

    ~~~ini [testenv:bashate] commands = bash -c 'find citellus -name "*.sh" -type f -print0 | xargs -0 bashate -i E006' ~~~

This change makes that execution of tox also pulls the output of bashate so all the integration already done for CI, was automatically update to do bash formatting too :-)

Bash i18n

Another topic that was interesting is the ability to easily write code in one language and via poedit or equivalent editors, be able to localize it.

In python is more or less easy as we did for citellus code, but I wasn't aware of any way of doing that for bash scripts (such as the plugins we do use for citellus).

Doing a simple man bash gives some hints somewhat hidden:

--dump-po-strings
    Equivalent to -D, but the output is in the GNU gettext po (portable object) file format.

So, bash has a way to dump 'po' strings (to be edited with poedit or your editor of choice), so only a bit more search was required to find how to really do it.

Apparently is a lot easier than I expected, as long as we take some considerations:

  • LANG shouldn't be C as it disables i18n
  • Environment variable TEXTDOMAIN should indicate the filename containing the translated strings.
  • Environment variable TEXTDOMAINDIR should contain the path to the root of the folder containing the translations, for example:
    • TEXTDOMAIN=citellus/locale
    • And language file for en as:
      • citellus/locale/en/LC_MESSAGES/$TEXTDOMAIN.mo

Now, the "trickier" part was to prepare scripts...

# Legacy way
echo "String"
# i18n way
echo $"String"
# Difficult... isn't it?

This change makes 'bash' to lock for the string inside $TEXTDOMAINDIR/locale/$LANG/LC_MESSAGES/$TEXTDOMAIN.mo and do on the fly replacement of the strings for the translated ones (or fallback to the one echoed).

In citellus we did implement it by exporting the extra variables defined above, so scripts, as well as framework is ready for translation!.

Just in case, some remarks: - I found some complains when same script outputs the same string in several places, what I did, is to create a VAR and echo that var.

  • As we've strings in citellus.py, magui.py, etc and the bash files, I did update a script to extract the required strings:
# Extract python strings
python setup.py extract_messages -F babel.cfg -k _L
# Extract bash strings
find citellus -name "*.sh" -exec bash --dump-po-strings "{}" \; > citellus/locale/citellus-plugins.pot
# Merge bash and python strings
msgcat -F citellus/locale/citellus.pot citellus/locale/citellus-plugins.pot > citellus/locale/citellus-new.pot
# Move file to destination
cat citellus/locale/citellus-new.pot > citellus/locale/citellus.pot

In this way, we're ready to use on editor to translate all the strings for the whole citellus + plugins.

Enjoy!

Click to read and post comments
Next → Page 1 of 2