Using Hudson CI server to build multiple projects with FlexUnit4 tests on a windows box in parallel

June 29th, 2010

Recently I setup the Hudson CI server for my splink projects. As Hudson is very well documented and super-easy to use, I had it up and running within less than 5 minutes.

1. Download the latest Hudson.war
2. Run Hudson "java -jar hudson.jar"
3. Install Hudson as a windows service

My next step was to add build jobs for the splink projects. For each project (splinklibrary, splinkresource and deepsplink I did

1. Add a "Free Style" job
2. Configure the source code repository (firstly I had to install the Hudson Mercurial plugin, but for instance SVN is supported out of the box.
3. Define when to run the job (i.e. every hour, after a commit, after another build, ...)
4. Add ANT scripts and define which tasks are to be run. (i.e. compile, run-tests, package, release, ...)
5. Define what to do after the ANT scripts have been successfully run. (i.e. publish documentation, publish test results, archive releases, ...)
6. Save the job

Now I could click the 'build button' and Hudson successfully built my project. Great!

But wait,..

Next I configured the three jobs to run every hour. But almost always one of the jobs failed. After looking at the console output of the failed builds I discovered the problem:

The FlexUnit tasks communicate with the swf which runs the FlexUnit tests over a socket on port 1024. So if two jobs run at a time, they interfere with each other because both jobs try to use port 1024 when they run the FlexUnit tests.

So I needed to assign unique TCP ports to avoid these port collisions and the Hudson Port Allocator Plug-in seems the best tool for the job. Just install it here
With the port allocator in place I could allocate one or more ports for a job. My first solution was to just define port 1024 for each job and be done. Port Allocator then queues all the jobs which use port 1024 instead of running them in parallel. On the upside: No more failed builds. But on the downside the builds take longer, as they can't execute in parallel anymore.
The answer to this was to make the Hudson Port Allocator Plug-in pick a free random port and convey the port to the build script. Because if each job is assigned with it's own port for the FlexUnit socket communication, jobs can't jam each other anymore.

Make Hudson Port Allocator convey a free port to the build script

Now I could grab the PORT enviroment variable in my build file and use it to configure the Hudson ANT task.

 
<property environment="env"/>
<condition property="PORT" value="${env.PORT}" else="1024">
	<isset property="env.PORT" />
</condition>
 
<flexunit
	swf="${tests}/TestRunner.swf"
	toDir="${report}"
	haltonfailure="false"
	headless="false"
	verbose="true"
	port="${PORT}"
	localTrusted="true">
</flexunit>
 

But as the FlexUnit CIListener class also needs to know the port, I employed ANT's echo task to write the port number into a file:

 
<echo file="${tests}/port">${port}</echo>
 
 
core = new FlexUnitCore();
core.addListener(new CIListener(port));
 

And to get the port inside my TestRunner class I had to just load the "port" file:

 
public function TestRunner() {
	var loader : QUrlLoader = new QUrlLoader(new URLRequest("port"));
	loader.register(QEvent.COMPLETE, function(e : QEvent):void {
		start(loader.getContent());
	});
	loader.register(QEvent.ERROR, function(e : QEvent):void {
		start();
	});
	loader.start();
}
 
private function start(port : uint = 1024) : void {
	core = new FlexUnitCore();
	core.addListener(new TraceListener());
	core.addListener(new CIListener(port));
	core.run(IntegrationSuite);
}
 

* QUrlLoader is part of the splinklibrary project.

By checking the console output after running a build job I was able to verify that Hudson Port Allocator indeed picked different ports for each job:

hudson-allocating
hudson-allocating-2


No Comments »




splinklibrary 0.2.0 and QLog 1.1 releases

September 16th, 2008

I just relased a new version of splinklibrary. There are a few bugfixes, some performance improvements and several new features:

- the logging framework is now prepared to address the new QLog features "tabbed logging" and "navigatable tree tabs"; If you use QLog you can send the current flash displaylist or the structure of any object to QLog which opens a new tab containing a navigatable tree of the displaylist or the object.

- the tween engine is now more than twice as fast, flexible and extendable
- there is a new "tree" package which makes working with tree structures quite convenient. For instance it is used internally to serialize the flash displaylist or any object into xml which can be sent via QLogOutput to QLog.

For more details see the splinklibrary svn changelog
To download the splinklibrary-0.2.0 release directly, click here or you can get it from the google code svn

QLog now supports a new kind of tab, the "tree tab" which is capable of displaying any xml structure sent by a client as a navigatable tree. Note that you need splinklibrary-0.2.0 in order to use the "tree" feature. You can grab the QLog 1.1. release here


No Comments »




QLog update

September 1st, 2008

I just added another couple of features to QLog and also made some major refactorings under the hood.
Now Qlog enables you to change the font size, it provides keyboard shortcuts for everything, there is a new useful scroll-lock toggle button which helps if you want to read the logs while new log messages arrive. QLog also exports your logfiles now as valid xhtml 1.0 strict files and saves all settings like window position, size and various other settings on exit, to spare you from the pain to adjust the settings each time you start QLog. Furthermore it enhances working with log tabs by highlighting tabs which are currently not focused but contain unread messages.

Get the latest QLog version here


No Comments »




New QLog release

August 27th, 2008

hi folks,

I recently became aware of a couple of features I really needed in QLog so I finally sat down to implement them.

Because I am currently working on an application which consits of several modules, each with it's own logger, the view of all the modules log messages in one window became quite cumbersome to read. So I wanted to see the logs of the modules each in a seperate tab to be able to better distinguish between them. Today I finally implemented this feature and QLog is now capable of creating different tabs for each logger connection. Thanks to the new tabs the information is far better structured and reading logs from different modules is now quite convenient.

Sometimes it's handy to save the logs to a file. That could be for instance to be able to compare the logs later, or simply to send them to someone else.  This can now easily be done through QLog's new save file feature, which renders a html file for the currently opened log tab.

I also discovered and fixed a bug which occurred only with the flash player versions 9,0,124,0 or above, because these and future flash player versions need the server to send a socket policy file to the client in order to establish a connection. read more

Well, that's it. You can download the latest QLog release here.

Hopefully fsteeg will add the new features to his eclipse plugin version of QLog soon.


No Comments »




AsBeanGen – Generate value objects for DTD driven XML files

June 28th, 2008

Update:

Download AsBeanGen tool
Download AsBeanGen source

I fixed some of the bugs mentioned in the comments:

- stripping '-' and '_' chars from the bean names
- handeling recursive DTD
- added 'implicit getters' mode
- new option to generate interfaces

.....................................................................................................................................................

AsBeanGen is a class generator application written in java which generates as3 data bean classes from DTD files. Overall it shortens development time a lot, because it rids you of spending a considerable amount of time on writing data bean classes.
Instead AsBeanGen can generate hunderts of classes in less than a second for you and as an added bonus it doesn't just generate the bean classes, but also the classes which transform the content of an XML file (implementing the DTD) to data bean objects. So finally you just have to write two lines of code yourself to get a strongly typed object representation of an XML file.

If you never heard of DTD (Document Type Definition), I suggest you to read this.
As DTD doesn't support type information natively, you also have to annotate the DTD files through simple comments to add the neccessary type information, otherwise AsBeanGen uses 'Object' as the standard type.

To give you a more specific idea how it is used, let's jump straight into an example:

This is the annotatd DTD file

 
<?xml version="1.0" encoding="UTF-8"?>
 
<!ELEMENT test (size,fonts+) >
 
	<!--
		Sets the types for the properties of the SizeData class 
 
		TYPE size.width=int
		TYPE size.height=int
	-->
	<!ELEMENT size EMPTY >
	<!ATTLIST size
		width CDATA #REQUIRED
		height CDATA #REQUIRED
	>
 
	<!--
		Sets the types for the properties of the FontsData class
 
		TYPE fonts.path=String
 
		Generates 'getFontDataById(id:String):FontData' method into
                FontsData class
		GENERATE font.id
 
		Generates 'getFontDataByName(name:String):FontData' method
                into FontsData class
		GENERATE font.name
	-->
	<!ELEMENT fonts (font+) >
		<!ATTLIST fonts
			path CDATA #REQUIRED >
 
			<!--
				Sets the types for the properties of the
                                FontData class
 
				TYPE font.id=String
				TYPE font.type=String
				TYPE font.name=String
				TYPE font.src=String
				TYPE font.systemfont=Boolean
			-->
			<!ELEMENT font EMPTY >
				<!ATTLIST font
					id CDATA #REQUIRED
					type CDATA #REQUIRED
					name CDATA #REQUIRED
					src CDATA #REQUIRED
					systemfont (true|false) #IMPLIED
				>
 

This is an xml file based on the DTD

 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE test SYSTEM "test.dtd">
 
<test>
	<size height="100" width="100"/>
	<fonts path="fonts/">
		<font id="foo" name="Arial" src="arial.swf" type="text"/>
		<font id="foo2" name="Arial2" src="arial2.swf" type="text2"/>
	</fonts>
</test>
 

This is the generated FontsData bean class

package data.test
{
 
/**
 * Class generated by AsBeanGen
 * http://www.splink.org
 */
 
	import data.test.fonts.FontData;
 
	public class FontsData
	{
		private var _path : String;
		private var _fontData : Array;
 
		public function FontsData()
		{
		}
 
		public function getPath() : String
		{
			return _path;
		}
 
		public function setPath(value:String) : void
		{
			_path = value;
		}
 
		public function getFontDataById(id:String) : FontData
		{
			var data:FontData;
			var r:FontData = null;
			for (var i:int=0; i&lt;_fontData.length; i++)
			{
				data = _fontData[i];
				if(data.getId() == id)
				{
					r = data;
					break;
				}
			}
			return r;
		}
 
		public function getFontDataByName(name:String) : FontData
		{
			var data:FontData;
			var r:FontData = null;
			for (var i:int=0; i&lt;_fontData.length; i++)
			{
				data = _fontData[i];
				if(data.getName() == name)
				{
					r = data;
					break;
				}
			}
			return r;
		}
 
		/**
		* @return an array of FontData objects
		*/
		public function getFontDataArray() : Array
		{
			return _fontData;
		}
 
		/**
		* @param an array of FontData objects
		*/
		public function setFontDataArray(value:Array) : void
		{
			_fontData = value;
		}
 
	}
}
 

This is the generated FontsDataReader class for the FontsData bean class

package data.test
 
{
 
/**
 * Class generated by AsBeanGen
 * http://www.splink.org
 */
 
	import data.test.FontsData;
	import data.test.fonts.FontDataReader;
 
	public class FontsDataReader
	{
		private var _xml : XML;
 
		public function FontsDataReader(xml:XML)
		{
			_xml = xml;
		}
 
		public function read() : FontsData
		{
			var fontsData:FontsData = new FontsData();
 
			fontsData.setPath(_xml.@path);
 
			fontsData.setFontDataArray(readFontData());
 
			return fontsData;
		}
 
		private function readFontData() : Array
		{
			var rAr:Array = [];
			for each(var xml:XML in _xml['font'])
			{
				var reader:FontDataReader = new FontDataReader(xml);
 				rAr.push(reader.read());
			}
			return rAr;
		}
	}
}
 

This is what the generated structure looks like:

This is how to use the generated classes in your application:

 
// TestDataReader is the root reader and expects the loaded xml file
// as a parameter.
var reader:TestDataReader = new TestDataReader(xml);
// TestDataReader's read method returns the root object of the
// generated tree, filled with values from the given xml.
var data:TestData = reader.read();
 

AsBeanGen can be used as a commandline application (usage shown below), or, if you are eclipse user you can add it as an 'External Tool' which proves to be very convenient (at least to me).

java -jar AsBeanGen.jar
test.DTD C:\your\project\src de.your.package.name.test

Note that the "test" in the package name is the name of the DTD file.

How to add AsBeanGen as an 'External Tool' in Eclipse

1) Open your annotated DTD file
2) Open the 'External Tools'



3) Create a 'New Launch Configuration'
4) Fill the form (see screenshot)

4) After configuring AsBeanGen, you will want it to appear in the External Tools favorites. Switch to the 'Common' tab and check 'External Tools' within the 'Display in favorites menu' box.
5) Press 'Apply' button

Now you should be able to launch AsBeanGen from the 'External Tools Favorites Bar'. Just make sure that the DTD from which you wish to generate the actionscript classes is opened and focused.

Download AsBeanGen


18 Comments »




QLog + splinklibrary Logging

May 30th, 2008

QLog is a java application which simply displays log messages. It enables to apply a filter to these messages so you can seek for a specific log or just display log messages which apply to the filter.

Because QLog is in fact a socket server it is also possible to display logs sent from an application which runs on a remote server. (This can be very handy). The greatest feature in my opionion is that messages sent to QLog can be equipped with a color for each kind of message (that is trace,info,debug,warn,error,fatal,...), so the messages look well arranged.

QLog view

In the whole QLog is a lot like powerflashers superb SocketOutputServer SOS but in contrast to SOS it is platform independent, it also runs on Mac Os X or Linux. (I developed it because i wanted to migrate to Ubuntu and I simply couldnt live without something like SOS)

QLog also integrates very well with the logging framework from splinklibrary. The splinklibrary logging framework comes with various ILogOutput implementations, so you can configure to which output destinations your logs are sent.
splinklibrary supports TraceOutput, FirebugOutput, SosOutput and QLogOutput out of the box. So you dont need to worry about any low level socket implementation details if you use QLog with splinklibrary.
One very useful feature of the sprinklibrary logging system is that if you run your application within the flash debug player, it adds the class- and methodname where the log originated from and if compiled with the -debug=true flag also the line number.

 
 
/**
 The Logger is configured once in a project. Each class which wants to
 log simply needs to add something like:
 private static const _logger:ILogger =
 LoggerProvider.getLogger("default", TheClassName);
 
*/
private function configureLogger():void
{
	var factory:ILoggerFactory = new LoggerFactory();
 
        /**
           It's possible to set various logger factories, so each one
           needs its own id
        */
	factory.setId("default");
 
        /**
         Logs within the specified range are sent to the log outputs
        */
	factory.setRange(new LogRange(LogLevel.TRACE, LogLevel.FATAL));
        /**
           DefaultOutputFormatter is used to format the log messages
           You can easily write and use your own IOutputFormatter
        */
	factory.setOuputFormatter(new DefaultOutputFormatter());
 
        /**
          Use different output strategies, here QLog and trace
          splinklibrary offers QLog, SOS, trace and Firebug LogOutput
          implementations. If you need something else, just implement
          the ILogOutput interface
        */
	factory.addLogOutput(new QLogOutput());
	factory.addLogOutput(new TraceOutput());
 
	LoggerProvider.addLoggerFactory(factory);
 
        /**
          Get a Logger and log something
        */
	_logger = LoggerProvider.getLogger("default", SampleApplicationClass);
        _logger.log(LogLevel.INFO, "Info");
        _logger.log(LogLevel.DEBUG, "Debug");
        _logger.log(LogLevel.ERROR, "Error");
}
 

Of course it should be quite simple to use Qlog with other Logging frameworks, or just to write a simple class to use it. splinklibararys QLogOutput should give a good impression how to do it.

Currently there is a stable QLog standalone version
Thanks to Fabian there is also a Eclipse Plugin version (very alpha) Update site: http://quui.com/update-site/
Here is a simple log4j appender class which enables to use QLog with the apache log4j framework for java.


No Comments »




DuDe

May 24th, 2008

Recently I discovered an interesting tool called DuDe. The author Richard Wettel describes DuDe as a 'text-based, language-independent code duplication (code clones) detector, gravitating around the concept of duplication chain.'


It is pretty simple to use; you start the jar, select a projects source folder and select the algorithm you want DuDe to use. Upon completion you can browse the results and learn which parts of your code could do with some optimizations.

I guess there are much more powerful tools for specific languages like java, but what makes DuDe shine brightly in my opinion is that it is language independent. So it works very well for actionscript source code or any other plain text files.


No Comments »




Powered by WordPress