My first steps into Ruby and OpenCV(on OSX)

I was writing some robotic code that will run on a mac mini, so I tryied on my macbook for the first time, ruby binding to opencv. It was a weird experience lol.

First, to install OpenCV, brew do the job:

$ brew tap homebrew/science
$ brew install opencv

after a while we see that instalation goes flawlessly:
/usr/local/Cellar/opencv/2.4.13.2: 278 files, 35.6MB
and we have current version: 2.4.13.2. That said, now install ruby gem:

$ gem install ruby-opencv -- --with-opencv-lib=/usr/local/Cellar/opencv/2.4.13.2/lib \
                             --with-opencv-include=/usr/local/Cellar/opencv/2.4.13.2/include/opencv \
                             --with-opencv-include=/usr/local/Cellar/opencv/2.4.13.2/include/opencv2

Fetching: ruby-opencv-0.0.18.gem (100%)
Building native extensions with: '--with-opencv-lib=/usr/local/Cellar/opencv/2.4.13.2/lib --with-opencv-include=/usr/local/Cellar/opencv/2.4.13.2/include/opencv --with-opencv-include=/usr/local/Cellar/opencv/2.4.13.2/include/opencv2'
This could take a while...
Successfully installed ruby-opencv-0.0.18
Parsing documentation for ruby-opencv-0.0.18
Installing ri documentation for ruby-opencv-0.0.18
Done installing documentation for ruby-opencv after 7 seconds
1 gem installed

after that, time to code.

My first try, obviously, is to use HAAR cascade classifier to look for faces.

require "rubygems"
require "opencv"
include OpenCV

window = GUI::Window.new("grab da face!")
camera = CvCapture.open
detector = CvHaarClassifierCascade::load('./haarcascade_frontalface_alt.xml')
loop {
  image = camera.query
  detector.detect_objects(image).each { |rect|
    image.rectangle! rect.top_left, rect.bottom_right, :color => CvColor::Blue
  }
  window.show image
  break if GUI::wait_key(100)
}

It didn’t worked as ruby-opencv currently supports only older type format of trained data xml. To solve that, I grabed older version of haarcascade_frontalface_alt.xml from https://raw.githubusercontent.com/Itseez/opencv/2.4.10.4/data/haarcascades/haarcascade_frontalface_alt.xml and it worked as expected.
Sometimes, OSX Facetime camera stop working, but is easy to fix, just run

sudo killall VDCAssistant

wait a little and voilá.

In the end, was a different experience, it worked but was way too slow. I’ll try to tweak a little but python version are way faster and I believe I’ll keep using it.

Advertisements

Accessing Go compiled applications through FFI

I finally started to create some useful Go code and, just like some previous posts(in pt-BR, ) where I integrated python, lua and C, I would love to use them with old python code instead of C/C++. For that, I had to generate a shared library. The process is easy as expected, requiring only the import of library “C” and a comment before function definition, exporting the function name, as following examples.

package main

import "C"

//export ModXY
func ModXY(x int) int {
        return x * 2
}

func main() {}

Now, let’s compile it to generate a shared lib:

$ go build -o libmod.so -buildmode=c-shared

This results on two files, libmod.so and libmod.h where the first one is a shared library itself and the second one is the headers to include Go types into a C application. More on this soon. First let’s check libmod.so

$ file libmod.so
libmod.so: Mach-O 64-bit dynamically linked shared library x86_64

Nice, as expected on OSX. Now let’s try to import it with python ctypes!

Python 2.7.12 (default, Sep 28 2016, 18:41:32)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> lib = ctypes.CDLL('libmod.so')
>>> lib.ModXY(13,2)
1

It worked like a charm! It is, actually, pretty easy to deal with. Now let’s find out the raison d’etre of this .h file. If, for some crazy reason, you need to embbed Go code inside a C file, simply include the .h and use the functions normally, as the following example:

#include <stdio.h>
#include "libmod.h"

int main(void){
  printf("the rest of division of 13 per 2 is %d", (int)ModXY(13,2));
  return 0;
}

then, compile it with:

$ gcc -o purexecutable testlib.c libmod.so 

and

$ file purexecutable
purexecutable: Mach-O 64-bit executable x86_64
$ ./purexecutable
the rest of division of 13 per 2 is 1

Yey! Again, simple and efficient. The design documentation of this feature may be found here( http://golang.org/s/execmodes ). That’s all

Let it go? Let it go!

Lately, I added Go as one of my favourite programming languages and I’ve tried to rewrite e create new projects in Go. Basically, the reasons Go became one of my favourite languages is that, although the language/compiler/tools is opensource, different from a majority of opensource projects (JS mainly), it is backed up by a huge corporation (Google) and is easy to avoid major mess (hello JS, my old friend). Also, Go is fast (really, really fast), simple (like C, not like python), concurrent and cross-platform by design, what makes Go a great general pourpose language. If all these previous reasons don’t catch your heart, Go is designed by Rob Pike and Ken Thompson, the same guys who designed UNIX and C, so there is no way this would be bad (fanboy attack?). Another very specific issue I had with c++ and libboost: take 3-8 hours compiling libraries whose Go version took 5 seconds, made me embrace Go as a friend.

I saw Go, from very begining, as a “pythonic C” and, for a python guy’s perspective, I loved it. And, being able to handle binary data just like C but structured data just like python, made me fall in love.

In the end, I started to use Go as a main language, I hope to gain more wisdom on it and become a better Gopher.

A study in software modification and automation by integrating Paterva’s CaseFile and Maltego

Maltego is an excellent intelligence and data visualization tool, but the need of being online(and the requirement of an registered account) make it pretty much useless for more sophisticated usages(like crime investigation or any other private and offline usage). For this, CaseFile was created, which is, basically, Maltego offline without the transforms. But transforms(offline and over closed-source data) would make the life of analysts easier in checking/validating information. To have the best of two worlds, I’ve extracted the transform engine from Maltego and inserted on CaseFile, also, removing the need of login and information leakage that the tool usually allow, making trustable and usable even by the government.

First, Maltego is a great tool and everyone who likes it should support Paterva and buy licenses. This is a study in software modification, automation and should not be used to cause harm to Paterva’s copyrights/busines model by any means. That’s why I’m using old versions of Maltego and Casefile.

The versions used was:

  • Maltego 3.1: maltego-3.1.1_CE-2012-04-11.zip md5 400b427652ca3e8ed60a6d6b7a457e81
  • CaseFile 1.0: maltego-CF.1.0.1_community-2012-03-14.zip md5 8d009eae5c899d74458712fe0e1458e1

They were originally downloaded from:

The base tools used was:

  • Jasmin ( http://jasmin.sourceforge.net/ ), an assembler for the Java Virtual Machine. It takes ASCII descriptions of Java classes, written in a simple assembler-like syntax using the Java Virtual Machine instruction set. It converts them into binary Java class files, suitable for loading by a Java runtime system.
  • ClassFileAnalyzer ( http://classfileanalyzer.javaseiten.de/ ), an analyzer and disassembler (Jasmin syntax 2) for Java class files.

First, I removed the annoying background image at maltego/modules/locale/com-paterva-maltego-ui-graph_maltego.jar.

Then, I’ve copied some files related to transform from Maltego to CaseFile:

  from maltego-ui/
    com-paterva-maltego-transforms-standard
    com-paterva-maltego-transform-protocol-v2
    com-paterva-maltego-transform-manager
    com-paterva-maltego-transform-finder     # needed by com-paterva-maltego-transform-manager
    com-paterva-maltego-transform-discovery  # needed by com-paterva-maltego-transform-protocol-v2
    com-paterva-maltego-transform-runner     # needed by com-paterva-maltego-transform-protocol-v2
  from maltego-core-platform/
    com-paterva-maltego-typing               # for com.paterva.maltego.typing.TypeNameValidator

The next step was define a series of modifications and fine tunning on CaseFile:

  • remove savetoserver and fake transform
  • remove startpage website
  • remove server discovery from manage transforms toolbar
  • always use trivialurldisplay
  • make showURL a stub in trivialurldisplay
  • remove google-me and wikipedia-me actions
  • remove all discover transforms actions
  • remove lots of webbrowser actions
  • enable transform limit toolbar

For that, I’ve put all modifications in .diff, .jdiff or .java where .diff is applied by standard POSIX patch utility, .jdiff is recompiled with jasmine and .java is compiled with jdk’s javac and integrated into target application.

Instructions:

  • Run cfpatch:
    ./cfpatch maltego-3.1.1_CE-2012-04-11.zip maltego-CF.1.0.1_community-2012-03-14.zip CF-custom.zip
  • Extact CF-custom.zip to the place where you want to install the custom CaseFile.

Extra
Create an .java file with the header:

//target-contains: com/paterva/maltego/graph/MaltegoGraph.class
//filename: org/hopto/im/Test.java

to integrate a new piece of software on Casefile. Example:

//target-contains: com/paterva/maltego/graph/MaltegoGraph.class
//filename: org/hopto/im/Test.java

// the above lines are to indicate that this class will be added
// in the same JAR module that file com/paterva/maltego/graph/MaltegoGraph.class
// and that the original name of this file is org/hopto/im/Test.java

package org.hopto.im;
public class Test {
    public static void main(String[] args) {
        System.out.println("Hello");
    }
}

All the working scripts and tools can be found at: https://github.com/paoloo/casefile-extender

Python + SSL para aplicações simples

Precisei relembrar uma forma de gerar uma conexão SSL(não apenas a camada de aplicação) para um projeto. O python faz isso de forma muito fluida e simples. Não requer prática, tampouco habilidade haha.

Para gerar o certificado, primeiro geramos uma chave com o openssl. Depois a usamos para gerar um certificado. Uma forma direta e automatica(e insegura, ja que não ha dados no certificado gerado) de fazer isso é:

openssl genrsa -out key.pem
yes '' | openssl req -new -x509 -key key.pem -out crt.pem -days 36500 || exit 1
cat key.pem crt.pem > cert.pem
rm key.pem crt.pem

Lembrando que estou ignorando as entradas para gerar o cert. Este tipo de coisa é mais usado para automatizar e gerar um cert novo a cada sessão. Depois disto é necessario pegar o fingerprint da chave, assim:

openssl x509 -in cert.pem -md5 -fingerprint -noout | sed 's/^[^=]\+=//;s/://g' | tr 'A-Z' 'a-z'

Agora o servidor, extremamente simples, para apenas uma conexão e printar o dado enviado:

import socket, ssl
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('', 8081))
sock.listen(1)
newsock, fromaddr = sock.accept()
conn = ssl.wrap_socket(newsock, server_side=True, certfile='cert.pem', keyfile='cert.pem')
conn.setblocking(0)
while True:
    try:
        buf = conn.read(512)
        if buf == '':
            break
        else:
            print buf
    except:
        pass

O cliente é tão simples quanto o servidor. Só lembrar de pegar o fingerprint gerado e usar no client. Ele verifica e só loga no servidor se bater o fingerprint.

import socket, ssl, hashlib, time
CERT="fingerprint que voce pegou"
sock = socket.socket()
sock.settimeout(2)
sock.connect(('localhost', 8081))
conn = ssl.wrap_socket(sock)
if hashlib.md5(conn.getpeercert(True)).hexdigest() != CERT:
    conn.close()
    print 'CERT diferente do esperado, tentando me hackearrrrrrr'
else:
    print '200 OK'
    time.sleep(2)
    conn.write('CHUPA ESSA MANGA, WIRESHARK!1!!')
    print 'enviado'
    conn.close()

E é só isso, bem simples. É uma forma de encriptação e autenticação bem razoavel e resolve para a maioria das aplicações práticas contemporaneas.

link dos sources em : https://github.com/paoloo/simple-ssl

Usando dados do acelerômetro de um smartphone com o browser

Enquanto migrava meus nós de uma Rede de Sensores sem Fio baseada em 802.15.4 para “Internet of Things” baseado em 802.11, principalmente em aplicações voltadas a robótica e movimento, onde os dados do acelerometro eram requeridos, precisei constantemente usar o Android Studio ou o APPinventor do MIT, e isso é algo contraproducente. Procurando alternativas mais diretas, acabei conhecendo o evento ‘devicemotion’ do javascript, e isso tornou as coisas muito mais divertidas!
Saber se o evento é suportado(tentei em muitos devices e é em todos) é bem fácil:

if (window.DeviceMotionEvent) {
  document.getElementById("INFO").innerHTML = "Acelerometro suportado"
} else {
  document.getElementById("INFO").innerHTML = "Acelerometro não suportado"
}

E usá-lo é ainda mais fácil:

  window.addEventListener('devicemotion', dmHandler, false);

O evento é hookado e chama uma função, dmHandler no caso, para tratá-lo. Uma exemplo de um handler seria:

function dmHandler(eventData) {
  acceleration = eventData.accelerationIncludingGravity;
  var left = 0; var right = 0;
  if (Math.abs(acceleration.y) > 1) {
    var speed = acceleration.y * 123;
    left = Math.min(1023, speed + acceleration.x * 100);
    right = Math.min(1023, speed - acceleration.x * 100);
  } else if (Math.abs(acceleration.x) > 1) {
    var speed = Math.min(1023, Math.abs(acceleration.x) * 123);
    if (acceleration.x > 0) {
      left = speed; right = -speed; 
    } else {
      left = -speed; right = speed;
    }
  }
  var direcao = "";
  direcao = "[" + Math.round(acceleration.x) + "," + Math.round(acceleration.y) + "," + Math.round(acceleration.z) + "]<BR/>" + Math.round(left) + ", " + Math.round(right); 
  document.getElementById("MOSTRA").innerHTML = direcao;
}

Depois disso, so chamar uma outra função para transferir os dados para o dispositivo:

var _ultimo = 0;
function diz(_esquerda, _direita) {
  var now = Date.now();
  if (_ultimo + 200 < now) {
     _ultimo = now; 
     var _req = new XMLHttpRequest();
     _req.open('GET', '/go/' + Math.round(_esquerda) + "," + Math.round(_direita), true);
     _req.send(null);
  }
}

E colocando tudo junto:

<!DOCTYPE HTML>
<html><head>
</head><body>
<div id="INFO"></div>
<br/>
<div id="MOSTRA"></div>
<br/>
<script type='text/javascript'>
if (window.DeviceMotionEvent) {
  window.addEventListener('devicemotion', dmHandler, false);
  document.getElementById("INFO").innerHTML = "Acelerometro suportado"
} else {
  document.getElementById("INFO").innerHTML = "Acelerometro nao suportado"
}
var _ultimo = 0;
function diz(_esquerda, _direita) {
  var now = Date.now();
  if (_ultimo + 200 < now) {
     _ultimo = now; 
     var _req = new XMLHttpRequest();
     _req.open('GET', '/go/' + Math.round(_esquerda) + "," + Math.round(_direita), true);
     _req.send(null);
  }
}

function dmHandler(eventData) {
  acceleration = eventData.accelerationIncludingGravity;
  var left = 0; var right = 0;
  if (Math.abs(acceleration.y) > 1) {
    var speed = acceleration.y * 123;
    left = Math.min(1023, speed + acceleration.x * 100);
    right = Math.min(1023, speed - acceleration.x * 100);
  } else if (Math.abs(acceleration.x) > 1) {
    var speed = Math.min(1023, Math.abs(acceleration.x) * 123);
    if (acceleration.x > 0) {
      left = speed; right = -speed; 
    } else {
      left = -speed; right = speed;
    }
  }
  var direcao = "";
  direcao = "[" + Math.round(acceleration.x) + "," + Math.round(acceleration.y) + "," + Math.round(acceleration.z) + "]<BR/>" + Math.round(left) + ", " + Math.round(right); 
  document.getElementById("MOSTRA").innerHTML = direcao;
}
</script>
</body></html>

E no meu embarcado, estou usando o bottle( https://paoloo.wordpress.com/2015/11/19/bottle-py-a-solucao-de-um-problema-que-nao-deveria-existir/ ) para gerenciar:

# -*- coding: utf-8 -*-
from bottle import request, route, run, static_file

@route('/')
def get_BASEdata():
    return static_file("loader.html", root="./")

@route('/go/<d>')
def get_SENTdata(d):
    print d # ou faz algo mais util, como enviar os valores para o controlador dos motores
    return d

run(host='0.0.0.0', port=8080, debug=True)

E é isso. Simples, direto e não requer nada além de um browser!

Serviço de checagem de CPF em bash script+python

Analisando um app de iPhone(depois mostro o passo a passo e as ferramentas), me vi precisando escrever um teste rápido no bash com o curl, que é uma ferramente absolutamente excepcional, mas me ví com problemas deevido a geração do HMAC e fazer parsing do resultado(sed+awk+cut é meio doloroso ne).
Acabei fazendo algo meio hibrido assim:

#!/bin/bash
cpfHASH=`echo "$1 $2" | python -c "from hashlib import sha1; from hmac import new as hmac; import sys; v=sys.stdin.read().strip().split(' '); print hmac('Sup3RbP4ssCr1t0grPhABr4sil','%s%s' % (v[0],v[1]),sha1).digest().encode('hex')"`

saidA=`curl https://movel01.receita.fazenda.gov.br/servicos-rfb/v2/IRPF/cpf -s -k -H "token: $cpfHASH" -H "plataforma: iPhone OS" -H "dispositivo: iPhone" -H "aplicativo: Pessoa Física" -H "versao: 8.3" -H "versao_app: 4.1" -d "cpf=$1&dataNascimento=$2" 2>&1`

python -c "null=None; a=$saidA; print ''.join(['%s = %s\n' % (c,a[c]) for c in a])"

Ficou meio porco usar o python para gerar o hash e para fazer o parsing, mas resolveu o meu problema. A execução é simples, o chato é que requer, além do CPF, a data de nascimento. Mas futuramente integrarei uma ferramenta para conseguir esta informação ;D

$ ./cpf.sh 13326724691 14121947
horaConsulta = 20:49:31
exception = None
codigoRetorno = 00
mensagemRetorno = OK
dataIsncricao = anterior a 10/11/1990
digitoVerificador = 00
nome = DILMA VANA ROUSSEFF
dataEmissao = 20/11/2015
horaEmissao = 08:49:32
mensagemObito = None
anoObito = 0000
codigoControle = CCA9.63A0.C92D.1C0D
codSituacaoCadastral = 0
dataConsulta = 20/11/2015
descSituacaoCadastral = REGULAR
dataNascimento = 14/12/1947

O código se encontra em https://github.com/paoloo/servicos/blob/master/cpf.sh

o/