VisualNacro, a visual macro recorder for VNC. Alpha version

10 files changed, 1028 insertions, 0 deletions

diff --git a/VisualNaCro/AUTHORS b/VisualNaCro/AUTHORS
new file mode 100644
index 0000000..1081101
--- /dev/null
+++ b/VisualNaCro/AUTHORS
@@ -0,0 +1,2 @@
+Johannes Schindelin <Johannes.Schindelin@gmx.de>
+
diff --git a/VisualNaCro/ChangeLog b/VisualNaCro/ChangeLog
new file mode 100644
index 0000000..03bb9d2
--- /dev/null
+++ b/VisualNaCro/ChangeLog
@@ -0,0 +1,3 @@
+2005-01-13: Johannes Schindelin <Johannes.Schindelin@gmx.de>
+	* started the project
+
diff --git a/VisualNaCro/Makefile.am b/VisualNaCro/Makefile.am
new file mode 100644
index 0000000..202b511
--- /dev/null
+++ b/VisualNaCro/Makefile.am
@@ -0,0 +1,50 @@
+INTERFACE=nacro.h
+SRCS=nacro.c
+OBJS=nacro.o
+ISRCS=nacro_wrap.c
+IOBJS=nacro_wrap.o
+TARGET=nacro
+LIBS= @LIBVNCSERVERLIBS@ -lvncclient
+
+nacro_CFLAGS= @LIBVNCSERVERCFLAGS@
+
+SWIGOPT=
+
+EXTRA_DIST=autogen.sh $(INTERFACE) $(SRCS) $(ISRCS) recorder.pl
+
+all: $(LIBPREFIX)$(TARGET)$(SO)
+
+# the following is borrowed from SWIG
+
+SWIG= @SWIG@
+
+##################################################################
+#####                       PERL 5                          ######
+##################################################################
+
+# You need to set this variable to the Perl5 directory containing the
+# files "perl.h", "EXTERN.h" and "XSUB.h".   With Perl5.003, it's
+# usually something like /usr/local/lib/perl5/arch-osname/5.003/CORE.
+
+PERL5_INCLUDE= @PERL5EXT@
+
+# Extra Perl specific dynamic linking options
+PERL5_DLNK   = @PERL5DYNAMICLINKING@
+PERL5_CCFLAGS = @PERL5CCFLAGS@
+
+# ----------------------------------------------------------------
+# Build a Perl5 dynamically loadable module (C)
+# ----------------------------------------------------------------
+
+$(ISRCS): $(INTERFACE)
+	$(SWIG) -perl5 $(SWIGOPT) $(INTERFACE)
+
+$(OBJS): $(SRCS)
+	$(CC) -c -Dbool=char $(CCSHARED) $(CFLAGS) -o $@ $^ $(LIBVNCSERVERCFLAGS) $(INCLUDES) -I$(PERL5_INCLUDE)
+
+$(IOBJS): $(ISRCS)
+	$(CC) -c -Dbool=char $(CCSHARED) $(CFLAGS) -o $@ $^ $(INCLUDES) $(PERL5_CCFLAGS) -I$(PERL5_INCLUDE)
+
+$(LIBPREFIX)$(TARGET)$(SO): $(OBJS) $(IOBJS)
+	$(LDSHARED) $(OBJS) $(IOBJS) $(PERL5_DLNK) $(LIBS) -o $(LIBPREFIX)$(TARGET)$(SO)
+
diff --git a/VisualNaCro/NEWS b/VisualNaCro/NEWS
new file mode 100644
index 0000000..838bb62
--- /dev/null
+++ b/VisualNaCro/NEWS
@@ -0,0 +1 @@
+No News yet
diff --git a/VisualNaCro/README b/VisualNaCro/README
new file mode 100644
index 0000000..bafdbc2
--- /dev/null
+++ b/VisualNaCro/README
@@ -0,0 +1,86 @@
+This is VisualNaCro.
+
+DISCLAIMER: recorder.pl is not yet functional.
+
+What does it?
+
+	It is a Perl module meant to remote control a VNC server.
+	
+	It includes a recorder (written in Perl) to make it easy to
+	record a macro, which is just a Perl script, and which you can
+	modify to your heart's content.
+	
+	The most important feature, however, is that you can mark a
+	rectangle which the Perl script will try to find again when you
+	run it. Thus when you play a game and want to hit a certain button,
+	you just hit the Ctrl key twice, mark the button, and from then on,
+	all mouse movements will be repeated relative to that button, even
+	if the button is somewhere else when you run the script the next
+	time.
+
+	If you know Tcl Expect, you will recognize this approach. Only this
+	time, it is not text, but an image which is expected.
+	
+How does it work?
+
+	It acts as a VNC proxy: your Perl script starts its own VNC server.
+	The script now can intercept inputs and outputs, and act upon them.
+	In order to write a macro, start
+	
+		recorder.pl host:port my_macro.pl
+	
+	connect with a vncviewer of your choice to <host2>:23, where <host2>
+	is the computer	on which recorder.pl was started (not necessarily the
+	same as the VNC server!). Now your actions are recorded into
+	my_macro.pl, and the images you want to grep for will be saved as
+	my_macro-1.pnm,	my_macro-2.pnm, ...
+
+Why did I do it?
+
+	Because I could ;-)
+
+	No really, I needed a way to write automated tests. While there
+	exist a lot of OpenSource programs for web testing, I found none
+	of them easy to use, and for GUI testing I found xautomation.
+
+	Xautomation has this "visual grep" (or "graphical expect") feature:
+	given an image it tries to find it on the desktop and returns the
+	coordinates. Unfortunately, there is no easy way to record macros
+	with it, and it only works on X11.
+
+	As I know VNC pretty well, and there are VNC servers for every OS
+	and gadget, I thought it might be cool to have this feature to
+	control a VNC server.
+
+	Actually, it makes it even easier: with plain X11, for example, you
+	can not know where on the screen the action is if you don't check
+	the whole screen. This complex problem is beautifully addressed
+	in Karl Runge's x11vnc.
+
+	My main purpose is to run regression tests on different browsers,
+	which I can easily do by starting Xvnc and using VisualNaCro.
+
+How did I do it?
+
+	I wondered long about how to do it. I couldn't take the same approach
+	as xautomation: I cannot connect to the VNC server thousand times
+	per second. So I decided to create an interface of LibVNCServer/
+	LibVNCClient for use in a script language.
+
+	Fortunately, this task is made very, very easy by SWIG. As Perl
+	is one of my favorite script languages, I decided to use this.
+	But SWIG makes it easy to use the very same interface for other
+	popular languages, so you are welcome to port VisualNaCro to
+	the language of your choice!
+	
+Isn't it pronounced "Visual Macro"?
+
+	Yes. But I liked the Visual Na Cro play of acronyms. I'm sorry if
+	you don't find it funny.
+
+What's the license?
+
+	GPL. It is based on LibVNCServer/LibVNCClient, so it has to be.
+	If you want to port this package to use vncreflector, which has a
+	BSD license, go ahead.
+
diff --git a/VisualNaCro/autogen.sh b/VisualNaCro/autogen.sh
new file mode 100755
index 0000000..2d1645d
--- /dev/null
+++ b/VisualNaCro/autogen.sh
@@ -0,0 +1,55 @@
+#! /bin/sh
+# Run this to generate all the initial makefiles, etc.
+
+srcdir=`dirname $0`
+test -z "$srcdir" && srcdir=.
+
+DIE=0
+
+AUTOMAKE=automake-1.4
+ACLOCAL=aclocal-1.4
+
+($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || {
+        AUTOMAKE=automake
+        ACLOCAL=aclocal
+}
+
+(autoconf --version) < /dev/null > /dev/null 2>&1 || {
+	echo
+	echo "You must have autoconf installed to compile VisualNaCro."
+	echo "Download the appropriate package for your distribution,"
+	echo "or get the source tarball at ftp://ftp.gnu.org/pub/gnu/"
+	DIE=1
+}
+
+($AUTOMAKE --version) < /dev/null > /dev/null 2>&1 || {
+	echo
+	echo "You must have automake installed to compile VisualNaCro."
+	echo "Get ftp://sourceware.cygnus.com/pub/automake/automake-1.4.tar.gz"
+	echo "(or a newer version if it is available)"
+	DIE=1
+}
+
+if test "$DIE" -eq 1; then
+	exit 1
+fi
+
+(test -f $srcdir/nacro.h) || {
+	echo "You must run this script in the top-level VisualNaCro directory"
+	exit 1
+}
+
+if test -z "$*"; then
+	echo "I am going to run ./configure with no arguments - if you wish "
+        echo "to pass any to it, please specify them on the $0 command line."
+fi
+
+$ACLOCAL $ACLOCAL_FLAGS
+#autoheader
+$AUTOMAKE --add-missing --copy
+autoconf
+
+echo "Running ./configure --enable-maintainer-mode" "$@"
+$srcdir/configure --enable-maintainer-mode "$@"
+
+echo "Now type 'make' to compile VisualNaCro."
diff --git a/VisualNaCro/configure.ac b/VisualNaCro/configure.ac
new file mode 100644
index 0000000..4472329
--- /dev/null
+++ b/VisualNaCro/configure.ac
@@ -0,0 +1,248 @@
+dnl Process this file with autoconf to produce a configure script.
+dnl The macros which aren't shipped with the autotools are stored in the 
+dnl Tools/config directory in .m4 files.
+
+AC_INIT([VisualNaCro],[0.1],[http://libvncserver.sourceforge.net])
+AC_PREREQ(2.54)
+AC_CANONICAL_HOST
+AM_INIT_AUTOMAKE
+
+dnl Checks for programs.
+AC_CHECK_PROGS(SWIG,swig)
+AC_CHECK_PROGS(LIBVNCSERVERCONFIG,libvncserver-config)
+AC_PROG_CC
+AC_PROG_RANLIB
+AC_EXEEXT
+AC_OBJEXT
+
+LIBVNCSERVERCFLAGS=`libvncserver-config --cflags`
+LIBVNCSERVERLIBS=`libvncserver-config --libs`
+AC_SUBST(LIBVNCSERVERCFLAGS)
+AC_SUBST(LIBVNCSERVERLIBS)
+
+dnl Checks for header files.
+AC_HEADER_STDC
+
+dnl How to specify include directories that may be system directories.
+# -I should not be used on system directories (GCC)
+if test "$GCC" = yes; then
+    ISYSTEM="-isystem "
+else
+    ISYSTEM="-I"
+fi
+
+
+# Set info about shared libraries.
+AC_SUBST(SO)
+AC_SUBST(LDSHARED)
+AC_SUBST(CCSHARED)
+AC_SUBST(LINKFORSHARED)
+
+# SO is the extension of shared libraries `(including the dot!)
+AC_MSG_CHECKING(SO)
+if test -z "$SO"
+then
+	case $host in
+	*-*-hp*) SO=.sl;;
+	*-*-darwin*) SO=.bundle;;
+	*-*-cygwin* | *-*-mingw*) SO=.dll;;
+	*) SO=.so;;
+	esac
+fi
+AC_MSG_RESULT($SO)
+
+# LDSHARED is the ld *command* used to create shared library
+# -- "ld" on SunOS 4.x.x, "ld -G" on SunOS 5.x, "ld -shared" on IRIX 5
+# (Shared libraries in this instance are shared modules to be loaded into
+# Python, as opposed to building Python itself as a shared library.)
+AC_MSG_CHECKING(LDSHARED)
+if test -z "$LDSHARED"
+then
+	case $host in
+	*-*-aix*) LDSHARED="\$(srcdir)/ld_so_aix \$(CC)";;
+	*-*-cygwin* | *-*-mingw*)
+            if test "$GCC" = yes; then
+                LDSHARED="$CC -shared"
+            else
+                if test "cl" = $CC ;  then
+                    # Microsoft Visual C++ (MSVC)
+                    LDSHARED="$CC -nologo -LD"
+                else
+                    # Unknown compiler try gcc approach
+                    LDSHARED="$CC -shared"
+                fi
+            fi ;;
+	*-*-irix5*) LDSHARED="ld -shared";;
+	*-*-irix6*) LDSHARED="ld ${SGI_ABI} -shared -all";;
+	*-*-sunos4*) LDSHARED="ld";;
+	*-*-solaris*) LDSHARED="ld -G";;
+	*-*-hp*) LDSHARED="ld -b";;
+	*-*-osf*) LDSHARED="ld -shared -expect_unresolved \"*\"";;
+	*-sequent-sysv4) LDSHARED="ld -G";;
+	*-*-next*)
+		if test "$ns_dyld"
+		then LDSHARED='$(CC) $(LDFLAGS) -bundle -prebind'
+		else LDSHARED='$(CC) $(CFLAGS) -nostdlib -r';
+		fi
+                if test "$with_next_framework" ; then
+		    LDSHARED="$LDSHARED \$(LDLIBRARY)"
+		fi ;;
+	*-*-linux*) LDSHARED="gcc -shared";;
+	*-*-dgux*) LDSHARED="ld -G";;
+	*-*-freebsd3*) LDSHARED="gcc -shared";;
+	*-*-freebsd* | *-*-openbsd*) LDSHARED="ld -Bshareable";;
+	*-*-netbsd*)
+		if [[ "`$CC -dM -E - </dev/null | grep __ELF__`" != "" ]]
+		then
+			LDSHARED="cc -shared"
+		else
+			LDSHARED="ld -Bshareable"
+		fi;;
+	*-sco-sysv*) LDSHARED="cc -G -KPIC -Ki486 -belf -Wl,-Bexport";;
+	*-*-darwin*) LDSHARED="cc -bundle -undefined suppress -flat_namespace";;
+	*)	LDSHARED="ld";;
+	esac
+fi
+AC_MSG_RESULT($LDSHARED)
+# CCSHARED are the C *flags* used to create objects to go into a shared
+# library (module) -- this is only needed for a few systems
+AC_MSG_CHECKING(CCSHARED)
+if test -z "$CCSHARED"
+then
+	case $host in
+	*-*-hp*) if test "$GCC" = yes;
+		 then CCSHARED="-fpic";
+		 else CCSHARED="+z";
+		 fi;;
+	*-*-linux*) CCSHARED="-fpic";;
+	*-*-freebsd* | *-*-openbsd*) CCSHARED="-fpic";;
+	*-*-netbsd*) CCSHARED="-fPIC";;
+	*-sco-sysv*) CCSHARED="-KPIC -dy -Bdynamic";;
+	*-*-irix6*)  case $CC in
+		   *gcc*) CCSHARED="-shared";;
+		   *) CCSHARED="";;
+		   esac;;
+	esac
+fi
+AC_MSG_RESULT($CCSHARED)
+
+# RPATH is the path used to look for shared library files.
+AC_MSG_CHECKING(RPATH)
+if test -z "$RPATH"
+then
+	case $host in
+	*-*-solaris*) RPATH='-R. -R$(exec_prefix)/lib';;
+        *-*-irix*) RPATH='-rpath .:$(exec_prefix)/lib';;
+	*-*-linux*) RPATH='-Xlinker -rpath $(exec_prefix)/lib -Xlinker -rpath .';;
+	*)	RPATH='';;
+	esac
+fi
+AC_MSG_RESULT($RPATH)
+AC_SUBST(RPATH)
+
+# LINKFORSHARED are the flags passed to the $(CC) command that links
+# the a few executables -- this is only needed for a few systems
+
+AC_MSG_CHECKING(LINKFORSHARED)
+if test -z "$LINKFORSHARED"
+then
+	case $host in
+	*-*-aix*)	LINKFORSHARED='-Wl,-bE:$(srcdir)/python.exp -lld';;
+	*-*-hp*)
+	    LINKFORSHARED="-Wl,-E -Wl,+s -Wl,+b\$(BINLIBDEST)/lib-dynload";;
+	*-*-linux*) LINKFORSHARED="-Xlinker -export-dynamic";;
+	*-*-next*) LINKFORSHARED="-u libsys_s";;
+	*-sco-sysv*) LINKFORSHARED="-Bdynamic -dy -Wl,-Bexport";;
+	*-*-irix6*) LINKFORSHARED="-all";;
+	esac
+fi
+AC_MSG_RESULT($LINKFORSHARED)
+
+# This variation is needed on OS-X because there is no (apparent) consistency in shared libary naming.
+# Sometimes .bundle works, but sometimes .so is needed.  It depends on the target language
+
+# Optional CFLAGS used to silence compiler warnings on some platforms.
+
+AC_SUBST(PLATFLAGS)
+case $host in
+   *-*-darwin*) PLATFLAGS="-Wno-long-double";;
+   *) PLATFLAGS="";;
+esac
+
+#----------------------------------------------------------------
+# Look for Perl5
+#----------------------------------------------------------------
+
+PERLBIN=
+
+AC_ARG_WITH(perl5,[  --with-perl5=path       Set location of Perl5 executable],[ PERLBIN="$withval"], [PERLBIN=])
+
+# First figure out what the name of Perl5 is
+
+if test -z "$PERLBIN"; then
+AC_CHECK_PROGS(PERL, perl perl5.6.1 perl5.6.0 perl5.004 perl5.003 perl5.002 perl5.001 perl5 perl)
+else
+PERL="$PERLBIN"
+fi
+
+
+AC_MSG_CHECKING(for Perl5 header files)
+if test -n "$PERL"; then
+	PERL5DIR=`($PERL -e 'use Config; print $Config{archlib}, "\n";') 2>/dev/null`
+	if test "$PERL5DIR" != ""; then
+		dirs="$PERL5DIR $PERL5DIR/CORE"
+		PERL5EXT=none
+		for i in $dirs; do
+			if test -r $i/perl.h; then
+				AC_MSG_RESULT($i)
+				PERL5EXT="$i"
+				break;
+			fi
+		done
+		if test "$PERL5EXT" = none; then
+			PERL5EXT="$PERL5DIR/CORE"
+			AC_MSG_RESULT(could not locate perl.h...using $PERL5EXT)
+		fi
+
+		AC_MSG_CHECKING(for Perl5 library)
+		PERL5LIB=`($PERL -e 'use Config; $_=$Config{libperl}; s/^lib//; s/$Config{_a}$//; print $_, "\n"') 2>/dev/null`
+		if test "$PERL5LIB" = "" ; then
+			AC_MSG_RESULT(not found)
+		else
+			AC_MSG_RESULT($PERL5LIB)
+		fi
+    AC_MSG_CHECKING(for Perl5 compiler options)
+ 		PERL5CCFLAGS=`($PERL -e 'use Config; print $Config{ccflags}, "\n"' | sed "s/-I/$ISYSTEM/") 2>/dev/null`
+ 		if test "$PERL5CCFLAGS" = "" ; then
+ 			AC_MSG_RESULT(not found)
+ 		else
+ 			AC_MSG_RESULT($PERL5CCFLAGS)
+ 		fi
+	else
+		AC_MSG_RESULT(unable to determine perl5 configuration)
+		PERL5EXT=$PERL5DIR
+	fi
+else
+       	AC_MSG_RESULT(could not figure out how to run perl5)
+fi
+
+# Cygwin (Windows) needs the library for dynamic linking
+case $host in
+*-*-cygwin* | *-*-mingw*) PERL5DYNAMICLINKING="-L$PERL5EXT -l$PERL5LIB";;
+*)PERL5DYNAMICLINKING="";;
+esac
+
+AC_SUBST(PERL)
+AC_SUBST(PERL5EXT)
+AC_SUBST(PERL5DYNAMICLINKING)
+AC_SUBST(PERL5LIB)
+AC_SUBST(PERL5CCFLAGS)
+
+#----------------------------------------------------------------
+# Miscellaneous
+#----------------------------------------------------------------
+
+AC_CONFIG_FILES([Makefile])
+AC_OUTPUT
+
+dnl configure.in ends here
diff --git a/VisualNaCro/nacro.c b/VisualNaCro/nacro.c
new file mode 100644
index 0000000..7f1a874
--- /dev/null
+++ b/VisualNaCro/nacro.c
@@ -0,0 +1,455 @@
+#include <assert.h>
+#include <rfb/rfb.h>
+#include <rfb/rfbclient.h>
+
+#include "nacro.h"
+
+/* for visual grepping */
+typedef struct image_t {
+	int width,height;
+	char* buffer;
+} image_t;
+
+/* this is a VNC connection */
+typedef struct private_resource_t {
+	int listen_port;
+	rfbScreenInfo* server;
+	rfbClient* client;
+
+	uint32_t keysym;
+	rfbBool keydown;
+
+	int x,y;
+	int buttons;
+
+	image_t* grep_image;
+	int x_origin,y_origin;
+
+	enum { SLEEP,VISUALGREP,WAITFORUPDATE } state;
+	result_t result;
+} private_resource_t;
+
+/* resource management */
+
+#define MAX_RESOURCE_COUNT 20
+
+static private_resource_t resource_pool[MAX_RESOURCE_COUNT];
+static int resource_count=0;
+
+private_resource_t* get_resource(int resource)
+{
+	if(resource>=MAX_RESOURCE_COUNT || resource<0 || resource_pool[resource].client==0)
+		return 0;
+	return resource_pool+resource;
+}
+
+private_resource_t* get_next_resource()
+{
+	if(resource_count<MAX_RESOURCE_COUNT) {
+		memset(resource_pool+resource_count,0,sizeof(private_resource_t));
+		resource_count++;
+		return resource_pool+resource_count-1;
+	} else {
+		int i;
+
+		for(i=0;i<MAX_RESOURCE_COUNT && resource_pool[i].client;i++);
+		if(i<MAX_RESOURCE_COUNT)
+			return resource_pool+i;
+	}
+	return 0;
+}
+
+void free_resource(int resource)
+{
+	private_resource_t* res=get_resource(resource);
+	if(res)
+		res->client=0;
+}
+
+/* hooks */
+
+void got_key(rfbBool down,rfbKeySym keysym,rfbClientRec* cl)
+{
+	private_resource_t* res=(private_resource_t*)cl->screen->screenData;
+	
+	res->keydown=down;
+	res->keysym=keysym;
+	res->result|=RESULT_KEY;
+}
+
+void got_mouse(int buttons,int x,int y,rfbClientRec* cl)
+{
+	private_resource_t* res=(private_resource_t*)cl->screen->screenData;
+	
+	res->buttons=buttons;
+	res->x=x;
+	res->y=y;
+	res->result|=RESULT_MOUSE;
+}
+
+rfbBool malloc_frame_buffer(rfbClient* cl)
+{
+	private_resource_t* res=(private_resource_t*)cl->clientData;
+
+	if(!res->server) {
+		int w=cl->width,h=cl->height;
+		
+		res->client->frameBuffer=malloc(w*4*h);
+		
+		res->server=rfbGetScreen(0,0,w,h,8,3,4);
+		res->server->screenData=res;
+		res->server->port=res->listen_port;
+		res->server->frameBuffer=res->client->frameBuffer;
+		res->server->kbdAddEvent=got_key;
+		res->server->ptrAddEvent=got_mouse;
+		rfbInitServer(res->server);
+	} else {
+		/* TODO: realloc if necessary */
+		/* TODO: resolution change: send NewFBSize */
+		/* TODO: if the origin is out of bounds, reset to 0 */
+	}
+}
+
+void got_frame_buffer(rfbClient* cl,int x,int y,int w,int h)
+{
+	private_resource_t* res=(private_resource_t*)cl->clientData;
+	
+	assert(res->server);
+	
+	if(res->grep_image) {
+		/* TODO: find image and set x_origin,y_origin if found */
+	} else {
+		res->state=RESULT_SCREEN;
+	}
+	if(res->server) {
+		rfbMarkRectAsModified(res->server,x,y,x+w,y+h);
+	}
+
+	res->result|=RESULT_SCREEN;
+}
+
+/* init/shutdown functions */
+
+resource_t initvnc(const char* server,int server_port,int listen_port)
+{
+	private_resource_t* res=get_next_resource();
+	int dummy=0;
+
+	if(res==0)
+		return -1;
+
+	/* remember for later */
+	res->listen_port=listen_port;
+	
+	res->client=rfbGetClient(8,3,4);
+	res->client->clientData=res;
+	res->client->GotFrameBufferUpdate=got_frame_buffer;
+	res->client->MallocFrameBuffer=malloc_frame_buffer;
+	res->client->serverHost=strdup(server);
+	res->client->serverPort=server_port;
+	res->client->appData.encodingsString="raw";
+	if(!rfbInitClient(res->client,&dummy,0)) {
+		res->client=0;
+		return -1;
+	}
+	return res-resource_pool;
+}
+
+void closevnc(resource_t resource)
+{
+	private_resource_t* res=get_resource(resource);
+	if(res==0)
+		return;
+
+	if(res->server)
+		rfbScreenCleanup(res->server);
+
+	assert(res->client);
+
+	rfbClientCleanup(res->client);
+
+	res->client=0;
+}
+
+/* PNM (image) helpers */
+
+bool_t savepnm(resource_t resource,const char* filename,int x1,int y1,int x2,int y2)
+{
+	private_resource_t* res=get_resource(resource);
+	int i,j,w,h;
+	uint32_t* buffer;
+	FILE* f;
+	
+	assert(res->client);
+	assert(res->client->format.depth==24);
+
+	w=res->client->width;
+	h=res->client->height;
+	buffer=(uint32_t*)res->client->frameBuffer;
+
+	if(res==0 || x1>x2 || y1>y2 || x1<0 || x2>=w || y1<0 || y2>=h)
+		return FALSE;
+
+	f=fopen(filename,"wb");
+	
+	if(f==0)
+		return FALSE;
+
+	fprintf(f,"P6\n%d %d\n255\n",1+x2-x1,1+y2-y1);
+	for(j=y1;j<=y2;j++)
+		for(i=x1;i<=x2;i++) {
+			fwrite(buffer+i+j*w,3,1,f);
+		}
+	if(fclose(f))
+		return FALSE;
+	return TRUE;
+}
+
+image_t* loadpnm(const char* filename)
+{
+	FILE* f=fopen(filename,"rb");
+	char buffer[1024];
+	int i,j,w,h;
+	image_t* image;
+	
+	if(f==0)
+		return 0;
+	
+	if(!fgets(buffer,1024,f) || strcmp("P6\n",buffer)) {
+		fclose(f);
+		return 0;
+	}
+
+	do {
+		fgets(buffer,1024,f);
+		if(feof(f)) {
+			fclose(f);
+			return 0;
+		}
+	} while(buffer[0]=='#');
+
+	if(!fgets(buffer,1024,f) || sscanf(buffer,"%d %d",&w,&h)!=2
+			|| !fgets(buffer,1024,f) || strcmp("255\n",buffer)) {
+		fclose(f);
+		return 0;
+	}
+		
+	image=(image_t*)malloc(sizeof(image_t));
+	image->width=w;
+	image->height=h;
+	image->buffer=malloc(w*4*h);
+	if(!image->buffer) {
+		fclose(f);
+		free(image);
+		return 0;
+	}
+	
+	for(j=0;j<h;j++)
+		for(i=0;i<w;i++)
+			if(fread(image->buffer+4*(i+w*j),3,1,f)!=3) {
+				fclose(f);
+				free(image->buffer);
+				free(image);
+				return 0;
+			}
+
+	fclose(f);
+	
+	return image;
+}
+
+void free_image(image_t* image)
+{
+	if(image->buffer)
+		free(image->buffer);
+	free(image);
+}
+
+/* process() and friends */
+
+/* this function returns only if res->result in return_mask */
+result_t private_process(resource_t resource,timeout_t timeout_in_seconds,result_t return_mask)
+{
+	private_resource_t* res=get_resource(resource);
+	fd_set fds;
+	struct timeval tv,tv_start,tv_end;
+	unsigned long timeout=(unsigned long)(timeout_in_seconds*1000000UL);
+	int count,max_fd;
+
+	if(res==0)
+		return 0;
+
+	assert(res->client);
+
+	gettimeofday(&tv_start,0);
+	res->result=0;
+
+	do {
+		unsigned long timeout_done;
+
+		if(res->server) {
+			rfbBool loop;
+			do {
+				loop=rfbProcessEvents(res->server,res->server->deferUpdateTime);
+			} while(loop && res->result&return_mask==0);
+
+			if(res->result&return_mask!=0)
+				return res->result;
+
+			memcpy((char*)&fds,(const char*)&(res->server->allFds),sizeof(fd_set));
+			max_fd=res->server->maxFd;
+		} else {
+			FD_ZERO(&fds);
+			max_fd=0;
+		}
+		FD_SET(res->client->sock,&fds);
+		if(res->client->sock>max_fd)
+			max_fd=res->client->sock;
+
+		gettimeofday(&tv_end,0);
+		timeout_done=tv_end.tv_usec-tv_start.tv_usec+
+			1000000L*(tv_end.tv_sec-tv_start.tv_sec);
+		if(timeout_done>=timeout)
+			return RESULT_TIMEOUT;
+
+		tv.tv_usec=((timeout-timeout_done)%1000000);
+		tv.tv_sec=(timeout-timeout_done)/1000000;
+
+		count=select(max_fd+1,&fds,0,0,&tv);
+		if(count<0)
+			return 0;
+
+		if(count>0) {
+			if(FD_ISSET(res->client->sock,&fds)) {
+				if(!HandleRFBServerMessage(res->client))
+					return 0;
+				if(res->result&return_mask!=0)
+					return res->result;
+			}
+		} else {
+			res->result|=RESULT_TIMEOUT;
+			return RESULT_TIMEOUT;
+		}
+	} while(1);
+
+	return RESULT_TIMEOUT;
+}
+
+result_t process(resource_t res,timeout_t timeout)
+{
+	return private_process(res,timeout,RESULT_TIMEOUT);
+}
+
+result_t waitforanything(resource_t res,timeout_t timeout)
+{
+	return private_process(res,timeout,-1);
+}
+
+result_t waitforinput(resource_t res,timeout_t timeout)
+{
+	return private_process(res,timeout,RESULT_KEY|RESULT_MOUSE|RESULT_TIMEOUT);
+}
+
+result_t waitforupdate(resource_t res,timeout_t timeout)
+{
+	return private_process(res,timeout,RESULT_SCREEN|RESULT_TIMEOUT);
+}
+
+result_t visualgrep(resource_t res,const char* filename,timeout_t timeout)
+{
+	/* TODO: load filename and set res->grep_image to this image */
+	return private_process(res,timeout,RESULT_FOUNDIMAGE|RESULT_TIMEOUT);
+}
+
+/* this is an overlay which is shown for a certain time */
+
+result_t alert(resource_t resource,const char* message,timeout_t timeout)
+{
+	private_resource_t* res=get_resource(resource);
+	char* fake_frame_buffer;
+	char* backup;
+	int w,h;
+	result_t result;
+	
+	if(res->server==0)
+		return -1;
+
+	w=res->server->width;
+	h=res->server->height;
+	
+	fake_frame_buffer=malloc(w*4*h);
+	if(!fake_frame_buffer)
+		return -1;
+	memcpy(fake_frame_buffer,res->server->frameBuffer,w*4*h);
+	/* TODO: draw message */
+	
+	backup=res->server->frameBuffer;
+	res->server->frameBuffer=fake_frame_buffer;
+
+	result=private_process(resource,timeout,-1);
+	
+	res->server->frameBuffer=backup;
+	/* TODO: rfbMarkRectAsModified() */
+
+	return result;
+}
+/* inspect last events */
+
+keysym_t getkeysym(resource_t res)
+{
+	private_resource_t* r=get_resource(res);
+	return r->keysym;
+}
+
+bool_t getkeydown(resource_t res)
+{
+	private_resource_t* r=get_resource(res);
+	return r->keydown;
+}
+
+coordinate_t getx(resource_t res)
+{
+	private_resource_t* r=get_resource(res);
+	return r->x;
+}
+
+coordinate_t gety(resource_t res)
+{
+	private_resource_t* r=get_resource(res);
+	return r->y;
+}
+
+buttons_t getbuttons(resource_t res)
+{
+	private_resource_t* r=get_resource(res);
+	return r->buttons;
+}
+
+/* send events to the server */
+
+bool_t sendkey(resource_t res,keysym_t keysym,bool_t keydown)
+{
+	private_resource_t* r=get_resource(res);
+	return SendKeyEvent(r->client,keysym,keydown);
+}
+
+bool_t sendmouse(resource_t res,coordinate_t x,coordinate_t y,buttons_t buttons)
+{
+	private_resource_t* r=get_resource(res);
+	return SendPointerEvent(r->client,x,y,buttons);
+}
+
+/* for visual grepping */
+
+coordinate_t getoriginx(resource_t res)
+{
+	private_resource_t* r=get_resource(res);
+	return r->x_origin;
+}
+
+coordinate_t getoriginy(resource_t res)
+{
+	private_resource_t* r=get_resource(res);
+	return r->y_origin;
+}
+
diff --git a/VisualNaCro/nacro.h b/VisualNaCro/nacro.h
new file mode 100644
index 0000000..51df69e
--- /dev/null
+++ b/VisualNaCro/nacro.h
@@ -0,0 +1,101 @@
+#ifndef NACRO_H
+#define NACRO_H
+
+#ifdef SWIG
+%module nacro
+
+%{
+
+/* types used */
+
+/* 0=false, every other value=true */
+typedef int bool_t;
+
+/* a keysym: identical with ASCII for values between 0-127 */
+typedef unsigned char keysym_t;
+
+/* this can be negative, because of a new origin set via visual grep */
+typedef int coordinate_t;
+
+/* left button is 1<<0, middle button is 1<<1, right button is 1<<2 */
+typedef unsigned char buttons_t;
+
+/* this is sort of a "file descriptor" for the proxy */
+typedef int resource_t;
+
+/* the timeout, specified in microseconds, for process() and friends */
+typedef double timeout_t;
+
+/* the return values of process() and friends */
+typedef enum {
+	RESULT_TIMEOUT=1,
+	RESULT_KEY=2,
+	RESULT_MOUSE=4,
+	RESULT_SCREEN=8,
+	RESULT_FOUNDIMAGE=16
+} result_t;
+
+%}
+
+#endif // SWIG
+
+typedef int bool_t;
+typedef unsigned char keysym_t;
+typedef int coordinate_t;
+typedef unsigned char buttons_t;
+typedef int resource_t;
+typedef double timeout_t;
+typedef enum {
+	RESULT_TIMEOUT=1,
+	RESULT_KEY=2,
+	RESULT_MOUSE=4,
+	RESULT_SCREEN=8,
+	RESULT_FOUNDIMAGE=16
+} result_t;
+
+/* init/shutdown */
+
+resource_t initvnc(const char* server,int serverPort,int listenPort);
+void closevnc(resource_t res);
+
+/* run the event loop for a while: process() and friends:
+ * process() returns only on timeout,
+ * waitforanything returns on any event (input, output or timeout),
+ * waitforupdate() returns only on timeout or screen update,
+ * waitforinput() returns only on timeout or user input,
+ * visualgrep() returns only on timeout or if the specified PNM was found
+ * 	(in that case, x_origin and y_origin are set to the upper left
+ * 	 corner of the matched image). */
+
+result_t process(resource_t res,timeout_t seconds);
+result_t waitforanything(resource_t res,timeout_t seconds);
+result_t waitforupdate(resource_t res,timeout_t seconds);
+result_t waitforinput(resource_t res,timeout_t seconds);
+result_t visualgrep(resource_t res,const char* filename,timeout_t seconds);
+
+/* inspect last events */
+
+keysym_t getkeysym(resource_t res);
+bool_t getkeydown(resource_t res);
+
+coordinate_t getx(resource_t res);
+coordinate_t gety(resource_t res);
+buttons_t getbuttons(resource_t res);
+
+/* send events to the server */
+
+bool_t sendkey(resource_t res,keysym_t keysym,bool_t keydown);
+bool_t sendmouse(resource_t res,coordinate_t x,coordinate_t y,buttons_t buttons);
+
+/* for visual grepping */
+
+coordinate_t getoriginx(resource_t res);
+coordinate_t getoriginy(resource_t res);
+
+bool_t savepnm(resource_t res,const char* filename,coordinate_t x1, coordinate_t y1, coordinate_t x2, coordinate_t y2);
+
+/* this displays an overlay which is shown for a certain time */
+
+result_t alert(resource_t res,const char* message,timeout_t timeout);
+
+#endif
diff --git a/VisualNaCro/recorder.pl b/VisualNaCro/recorder.pl
new file mode 100644
index 0000000..023ae26
--- /dev/null
+++ b/VisualNaCro/recorder.pl
@@ -0,0 +1,27 @@
+#!/usr/bin/perl
+
+use nacro;
+
+$vnc=nacro::initvnc("localhost",5900,5923);
+
+print $vnc;
+
+# give it a chance to get a first screen update
+
+print nacro::waitforupdate($vnc,.4);
+
+print STDERR "Now\n";
+
+print nacro::sendmouse($vnc,90,250,0);
+
+print nacro::sendkey($vnc,ord('a'),-1);
+print nacro::sendkey($vnc,ord('a'),0);
+
+print nacro::sendmouse($vnc,100,10,0);
+
+print nacro::savepnm($vnc,"hallo.pnm",50,50,300,200);
+
+nacro::process($vnc,3);
+
+print"\n";
+