K++ Lab Insight

2012-02-10T18:03:00.000+01:00

KJSONPlusPlus

A clean, elegant and portable JavaScript Object Notation library for C++. This is a C++ library enabling programmers to generate and consume JSON data.

It's full source code is available here on github.

Features

The library is fully compliant with the JSON standard.
See http://www.json.org/

The library enables extended data types in addition to the standard ones:

DateTime in ISO-8601 format.
Arbitrary binary data encoded in Base85 format.
These extended data types are encoded inside the string standard data type. They are automatically recognized as extended types by the library. This makes the data generated by this library always standard-compliant. If the generated data is consumed by another JSON parser the data will still be readable and well-formatted.

The library is standalone. It only depends on C++ STL and C runtime.
The library's JSON scanner is automatically generated via RE2C.
See http://re2c.org/
This delivers a very fast and flexible JSON scanner.
The library's JSON parser is automatically generated via Bison. This enables a very clean JSON parser.
See http://www.gnu.org/s/bison/
See http://gnuwin32.sourceforge.net/
The library is hierarchically divided into modules of classes that further localize responsibilities.
JSON data source can be a file or a memory block or virtually any character-based data source.
In case of JSON format errors in the source data, the library parser reports the nature and location of errors: line/column of the error and a description of what was wrong.
JSON data can be encoded in UTF-8 or in Unicode.
Internally all string data is encoded in Unicode. Conversions happen only when needed, and happen automatically.
JSON data is loaded from the source into a hierarchy of objects in memory.
The representation is based on native C++ objects and primitive variables.
Representation persistence (saving back to a file) is fully supported and happens via a single method call.
Dynamic exploration and navigation in JSON data is possible.
No preliminary knowledge on data structure is required in order to use the data. However, preliminary knowledge of data structure can speed up data consumption and lower its complexity.
A JSON value can be treated generically as a value, or specifically given its data type.
i.e. An object can be treated merely as a value when it is to be deleted, and can be treated an an object value when one wants to explore its child values.
Many data conversions are automatically supported by the native objects in memory.
e.g. String value to integer, DataTime value to String,...
Generated JSON data is well-formatted and spaced and indented as required.
This allows for much better human-readability, but it reduces the parsing performance a little bit.
The library does not use C++ exceptions.
It reports errors via errno and returns values that indicate errno values.

Limitations

Progressive loading of JSON data is not supported.
ISO-8601 date & time format is not fully supported.
Only the most common form is supported. This should be enough for most usage cases.
No support for XPath-like access protocol.

Licence

Copyright 2011-2011 Koutheir Attouchi. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE FREEBSD PROJECT ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL KOUTHEIR ATTOUCHI OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of Koutheir Attouchi.

Installing and configuring Apache Axis C++ on Windows

2011-12-31T04:04:00.000+01:00

Briefly, I am sharing an experience with anyone interested in Apache Axis C++ installation and configuration.

I illustrate how to install Apache Axis C++ and configure the system for it, then how to get a working example of Calculator.

First, Four programs are to be installed:

Some files are to be downloaded from here. Once everything installed, we proceed to configuration.

GNUWin32

There is nothing special to do for this, but let's say it was installed in the folder T.

Apache HTTPD

Let's say it was installed in the folder Z. Open the text file "Z\conf\httpd.conf" and append the following text to end of it:

LoadModule axis_module modules/mod_axis2.so
<Location /axis>
   SetHandler axis
</Location>

Xerces

Extract the compressed file in a folder, let's say X.
Copy the file "X\bin\xerces-c_2_2_0.dll" on the desktop, we will use it later. You can delete the remaining extracted files.

Apache Axis C++

Axis will be installed in two folders on the machine; one for development (with the /include and /lib and others) and one for deployment of binaries of the Web services on the server side.

Extract the compressed file of Axis in a folder, let's say Y.
Rename the file "Y\bin\AxisXMLParserXerces.dll" to "Y\bin\AxisXMLParser.dll".
Move the file "xerces-c_2_2_0.dll" from the desktop to "Y\bin\xerces-c_2_2_0.dll".
Copy the files "MSVCP70.DLL" and "msvcr70.dll" from the desktop to "Y\bin\MSVCP70.DLL" and "Y\bin\msvcr70.dll" respectively. Il would also recommend copying those files to the system folder "Windows\System32".
Move the file "axiscpp.conf" to "Y\deploy\axiscpp.conf".
Create an empty folder "Y\deploy\repository".
Copy all the contents of the folder "Y\bin" to the folder "Y\deploy\lib".
Copy the folder "Y\deploy" in the folder "Z".
Rename the folder "Z\deploy" to "Z\Axis".
Copy the file "Z\Axis\lib\mod_axis2.dll" to the folder "Z\modules".
Rename the file "Z\modules\mod_axis2.dll" to "Z\modules\mod_axis2.so".
Open the file "Z\Axis\axiscpp.conf" and adapt it depending on the exact installation configuration.

Now we need to set some environment variables. Adapt those to your own installation configuration.

New variable: AXISCPP_DEPLOY=Z\Axis
Append to variable: PATH=...;T\bin;Z\bin;Z\Axis\lib

Now, the platform is ready. Let's experiment with an example.

Calculator example

The Calculator example is made up of multiple files for source code, and configuration, and description.
Some description files already exist in Axis for this trivial example. We will add the configuration file "server.wsdd" and some source files.

Copy the file "server.wsdd" to the folder "Z\Axis\conf".
Open the file "Z\Axis\axiscpp.conf" and adapt it to match the installation configuration. Particularly, the line: "WSDDFilePath:".
Copy the folder "Calculator" in your development folder/workspace.
Compile the Calculator server-side project to generate the binary "Calculator_Server.dll".
Adapt the file "Z\Axis\conf\server.wsdd" to the location of the file "Calculator_Server.dll".
Compile the Calculator client-side project to generate "Calculator.exe".

Until here, the example is compiled and deployed. Let's run it!

Start Apache HTTPD.
Using the Windows shell (DOS) or the IDE, run "Calculator.exe" with some parameters, and observe the results. For exemple:
"Calculator.exe mul 4 3 http://localhost:8080/axis/Calculator"
This prints "12" on the console: the result of running the web service on the server.
An abservation of "Z\logs\access.log" shows a treated request by HTTPD:
"POST /axis/Calculator HTTP/1.1" 200 400

Home-made Linux Backup Tool

2011-02-13T16:37:00.000+01:00

I know I know... Linux has many fully-featured backup tools able to backup your system, configuration, user files,... from different levels and to different mediums. But, in my situation, I had some quite special requirements for the backup tool I want to use:

It should backup the whole system as is, at the raw level. Backup partitions, not files. As I needed to get away from the file system support and different problems arising when backing up individual files.
It should enable compression of the data. I went a bit far to ask for 7zip compression levels ;-)
It should be smart enough to avoid backing up the free space on the partition. And here began some problems I will talk about later.
It should have a balance between optimal performance and backup safety.
It should give me the maximum configuration options. And the default configuration should be optimal for most cases.
It should depend on the least number of tools, so I can backup my Linux OS from a Live CD for example, without requiring installation of rare or fat utilities.
It should give some feedback on the progression of the lengthy operations.
It should not be too hard to code too!

So here is my point clear now. Let's move to the heart of it!
I created a shell script called full-backup-1.0.sh that did just what I wanted. I will present it here so the community takes advantage of it and help me to make it better or fix some bugs, if any.

Usage

full-backup-1.0.sh input_device [--no-fs-check] [--no-zero-free-space] [--no-compress] [--read-only] [--safe-rbs] [--rbs=RBS]

This will perform a full raw backup of the partition data and create a single file holding the data of the partition in the current directory.

input_device: Source partition device node.
--no-fs-check: Disable file system check before starting the backup. This is not recommended for safety reasons. By default, file system check is enabled.
--no-zero-free-space: Disable filling the empty space in the partition with zeros. This could be useful to be able to recover deleted files from the partition later, but can reduce the compression ratio of the backup. By default, filling the empty space in the partition with zeros is enabled.
--no-compress: Disable compression. This implies the option --no-zero-free-space. This will result in a file that has the same size as the partition. This is useful when backup time must be the least possible while backup size is not an issue.
--read-only: Disable any operations that can modify the partition. This implies the options --no-fs-check and --no-zero-free-space.
--safe-rbs: Use a safe read buffer size. The read buffer size will be set to 512 bytes. Read buffer size is the number of bytes that are read at once from the source partition. Safe here means that if the partition medium has some damaged sectors, then the bad sectors will not hide the readable ones. Use this if you are unsure about the health of the source partition medium or when performance is not an issue.
--rbs=RBS: Set the read buffer size to the specified RBS size. By default RBS is 64KB (65536 bytes) and this should be a good deal for most cases where partition medium health is good.

Example:
full-backup-1.0.sh /dev/sda2

Personal experience

Using this command, I backed up my Debian Linux partition (Total size: 32 GB, Used size: 8 GB) in a compressed file of 2 GB size in 2h25min. The backup was done from a shell running on a Knoppix Linux Live DVD. For me, this is exactly what I needed.

Download

You can download the script with 7zip binaries for Intel i386 architecture here. The file is "tar.gz"-compressed.

Implementation details

Here I present the shell script I wrote with details on each section of it.

Simple BSD License

As usual, I license this code under the terms of the Simple BSD License. Here is the contents of the license:

 Copyright (c) 2011 Koutheir Attouchi.
All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Internal state initialization

Here I initialize the contents of the global variables used in the script. Here is the meaning of them:

readonly: this will be a read-only operation.
checkfs: check the file system for errors before the backup.
zerofs: fill the empty space of the partition with zeros.
imp: input mount point for the source partition.
zf: the file full with zeros that will be created on the source partition.
of: output file name.
nocompress: do not compress the output file.
compressor: file name of the executable file which will compress the data.
pidfile: a file that will hold a process identifier to allow for inter-process communication between child processes launched by this script.
statusfile: a FIFO pipe file that will enable progression display.
tempfile: a FIFO pipe file that will enable input data size control.
uiupdatedelay: delay between progress display updates.
rbs: reading buffer size.

Command line parsing

Here is a regular command line parsing, nothing special to tell.

Preliminary checks

For the moment, a simple check whether the source partition device node exists.

Internal state adjustements

The extension of the backup output file is adjusted to reflect the nature of the file (compressed or not).
Read-only mode disables the file system check and filling the free space with zeros.

Reading source partition information

As a normal assumption, I am always backing up Linux partitions holding ext2 file system or later. This allows me the use of the dumpe2fs utility to gather some file system information.

Listing current backup configuration

This is done to tell what will be done next so the user knows in advance if he configured the tool as desired or not.

Unmounting partition

This is done because it's unreliable to backup a partition that is mounted, unless some more advanced technologies are used, which we avoid here.

Checking partition file system for errors

This is a security measure, and it is very recommended.

Filling partition free space with zeros

This is a technique used to allow for the best possible compression ratio of the partition, avoiding backing up the free space, while still being a simple technique.
It tries to fill the empty space on the source partition with a huge file filled with zeros. Once the partition is full, remove the file. This leaves the space previously allocated for the file with zeros (most of it).
Once done, this will higher the frequency of zero blocks and will allow the compressor to allocate very small space to represent the free space on the partition.
Operation progress will be shown during the filling operation which may take a long time to get done. Once the operation is done, the partition is unmounted again.
Once drawback for this method, it overwrites all the free space on the partition. Thus it renders impossible the recovery of previously deleted files on the partition. But if that is not an issue, one should be fine using this technique.

Backing up partition data

In uncompressed form

This uses the dd utility to backup the partition and removes any padding at the end of operation. The padding may be done due to the block size specified to dd.

In 7zip-compressed form

This uses two instances of dd: one to read the partition data and another to limit the output to the really read data, thus avoiding the compression of the padding bits.
The compression used here is the "Ultra" profile of 7zip compressor, which should produce the best possible compression ratio, but will consume much memory and much CPU power.
As 7zip is not a standard program, I put it beside the shell script so that I can use it from a live medium like Knoppix.

Finalization

This cleans up the temporary files and reports the status of the backup operation.

Minimize Firefox RAM Memory usage

2010-12-09T01:21:00.003+01:00

Introduction

Today, many of you are using the Firefox browser to surf on the Internet and develop web applications, etc... Once you begin using Firefox you may realize that it consumes too much RAM which goes up to 200MB in normal usage and can reach 700MB after several hours of continuous browsing. Yet, many of us still have a 1GB laptop or even a 512MB good old desktop computer. This means that if you use Firefox to browse on such computers, your multi-tasking ability is severely reduced.

This made me decide to put in action a program that reduces RAM usage of Firefox by forcing the OS to move the memory pages of Firefox to the swap space, thus leaving much more RAM available. And I quickly made it using the C++ language.

From the end, it cuts down Firefox RAM usage to about 15MB! Cool! No?
Well, it is really cool especially when you know that it have no impact on performance because of the paging operations the OS does. This is because Firefox is a network-based program, so the major latency factor is the network speed. And the loss of CPU power put in swapping the memory pages is very thin compared to the network latency. Of course you won't probably use my program to minimize the RAM usage of Oracle Database Server...

Features

Here are some of the features:

Actually, the program is able to reduce RAM usage for any program, and any number of programs, not only Firefox instances.
It does this in a safe way (no code injection or dirty hacks).
It's open source under the "Simple BSD Licence".
It is very tiny in disk size (less than 90KB) and in RAM usage (less than 300KB).
It consumes almost zero CPU usage with the default configuration options.
It has no GUI and no Console.
It can be configured using command line options or a simple configuration file. I recommend using configuration file for novice users.
It's behavior can be customized depending on usage situations. The default options should suit most needs.
It is portable and consists of only one executable binary.
Runs on Windows 2000 or later, including XP, Vista, 7, 2003 Server, 2008 Server.
It has a modular design which enables source code reuse.

Download

The program is called MinimizeRAMUsage. And here I present the binaries and source files for download:

Executable binary file: MinimizeRAMUsage.exe.
Source files archive: MinimizeRAMUsage.zip. This includes the Visual C++ 2010 solution and project files.

Usage

Now, to use the program, place it anywhere you want, then execute it. You will notice that nothing happens, except that a text file named "MinimizeRAMUsage.config.txt" appears next to the program in the same folder.
This file is the configuration file. Don't worry, open it using Notepad for example. You will find something like this:

timeout=400
rescan=1
#process=firefox.exe

To enable memory usage minimization for firefox, only remove the # and save the file. The contents of the file would become:

timeout=400
rescan=1
process=firefox.exe

Now execute the program again, and here it goes! You will find it in the Windows "Task Manager" and you will notice that Firefox RAM usage dropped instantly.

To make it do that automatically all the time,proceed as follows:
1. Right-Click on the MinimizeRAMUsage.exe program and choose "Create shortcut".
2. Cut the shortcut file created and paste it in the start up folder. This folder is located in: Start Menu > Programs > Startup. If you use the French version of Windows, you will find it in: Menu Démarrer > Programmes > Démarrage.
3. You're done, every time the PC boots, it will start the program and you will never ever again see Firefox consuming all your RAM memory.

Advanced usage

Now if you are an advanced or curious user, let me explain the configuration options:

"timeout=n" where n in an integer greater than or equal to 1. This is the duration in milliseconds that separates two periodic minimization routines done by the program. The lower it is the more you swap pages and the more free RAM you have and the higher CPU you consume. The higher it is, the lower CPU you consume and the lower free RAM you have.
"rescan=b" where b is one of {0, 1}. When this is 1, the program rescans the running processes every time before minimization routine, to ensure all instances of the given targets are operated on, even those started after the startup of MinimizeRAMUsage.exe. If this is 0, then the program will only operate on the instances that existed when it was started, and it will terminate its execution when all those end their execution.
"process=name.exe" where name.exe is the name of the executable whose RAM usage is to be minimized. This can be given several times and it operates in an incremental way.

And example of configuration file might be:

timeout=400
rescan=1
process=firefox.exe
process=chrome.exe
process=iexplore.exe

There is another way to configure this program: the command line options. The syntax is the following:
MinimizeRAMUsage.exe [-r] [-t timeout] file1.exe [file2.exe, [file3.exe[...]]]
Where:

-r option disables periodic rescan, which is enabled by default.
-t option indicates timeout in milliseconds, which is 1 by default.
file1.exe, file2.exe, file3.exe, ... are the executable programs to operate on.

The command line arguments overwrite the configuration file options.

That's all! I wish this come handy to some of you, as it is to me!

How to use more than 4GB of RAM with 32-bits Windows OS?

2010-12-08T04:18:00.003+01:00

Actually, I am still running the good old Windows XP 32-bits system on my 4GB RAM laptop. And as some of you may know, Windows 32-bits OS can use a maximum of 4GB of RAM due to the addressing space constraint.

In practice, my Windows XP only allows me to use a maximum of 3GB of RAM for my programs, while 1GB is restricted to the kernel. This, given that I enabled the PAE (Physical Address Extension) option for the kernel.

Anyway, I found a solution that enables me to use the whole 4GB for my applications, and that enables in general to use even more than 4GB of RAM with a 32-Bits Windows OS.

The trick is easy. Among the memories used by the system there is the RAM and the disk swap space (commonly C:\PageFile.sys) and other types also. I will let Windows use the 3GB of RAM as normal. Then I will make it use the last 1GB of RAM as a swap space, as if it was on disk. But to do this I will have to create a RAMDisk. A RAM disk is a virtual volatile disk that exist only in RAM. This enables me to regain the 1GB RAM space and use it for memory pagination operations. So nothing is lost.

To do so, I use a program called VSuite Ramdisk. This program enables me to create a new virtual disk drive in "My Computer" folder that enables me to access the 1GB RAM space that the OS cannot access natively.

You can download the program from here. The free edition does well. Once installed, you have to configure it to make the new RAM drive. Here is how:

1. Open the program.
2. Click on "Options" on the left tab bar.
3. Check the option "Enable OS Invisible Physical Memory".
4. Click on "Ramdisk" on the left tab bar.
5. Look at the option "Use OS Invisible Memory", next to it there is the total space of the RAM the 32-bits OS cannot use. Note down that number, I will called it OSIRS.
6. Choose the options for the new RAM drive. Personally I chose:
- Disk Size: 892MB
- Drive Letter: M:
- File System: FAT. You will need to use FAT32 if you have 6GB or more of RAM.
- Volume Label: RAMDISK
The rest of the options I kept for the default.
7. Click on "Add" button.
8. Click on "Exit" button.

Now you have a new drive in your computer called RAMDISK accessible as M:. Because of its volatile nature, this drive does not retain any files written into it once the computer is rebooted or shutdown.

Now, we will tell Windows to use our new drive as a container for its swap space. Here is the procedure:
1. Right click on "My Computer" and select "Properties".
2. Click on "Advanced" tab.
3. Click on "Settings" button inside "Performance" frame.
4. Click on "Advanced" tab.
5. Click on "Change" button inside "Virtual Memory" frame.
6. Select "M: RAMDISK" from the list.
7. Choose the option "Custom size" inside "Paging file size for selected drive" frame.
8. In "Initial size (MB)", type the number OSIRS minus 6. For example, in my case I type "886".
9. Type the same number in the text field "Maximum size (MB)".
10. Click the button "Set".
11. Click the button "OK".
12. Click the button "OK".
13. Click the button "OK".

Now, Windows will use the RAM disk every time it runs out of those 3GB RAM memory. Windows will not tell you that it uses the full 4GB (or maybe more in your case), but it will effectively do it.

Windows is still configured to use hard disk space to do paging operations, and because of this, some might argue that Windows might use the hard disk to swap pages of memory, instead of the RAM disk, which makes our trick meaningless, but here some ways to get around it:
1. You can remove the swap space from all other drives except the RAM disk. This will make dead sure Windows will use the RAM disk for memory page swapping. But this limits the virtual memory size to exactly 4GB, and in deed disables disk paging. This is not recommended.
2. Reduce the disk space used on other drives. This is my adopted solution. I limit Windows to only 1GB of hard disk space (on C:) in addition to the 886MB space of the RAM drive. This gives me about 5GB of virtual memory. By experience, I have never got in an out of memory situation since I bought my laptop (This means about 5 months of usage without a single out of memory situation). This proves that the 5GB are really sufficient for my jobs. The amount of hard disk space you need depends on your set of applications and how you use them. You can test for yourself to find a suitable configuration, you can always higher that size given that you have free space on the hard disk.

That's all. Thank you for your attention. I wish I have given something useful.

Installing ATI Proprietary Linux Driver (fglrx) on Debian Lenny 5.0 Stable 64 bits

2010-12-06T00:26:00.002+01:00

I am one of those unlucky peoples who happen to own a laptop equipped with an ATI graphic card, while being strongly attached to the open source world...

Yes, again, and the fight just continues between open source world (Linux, BSD, ...) and the ATI for its graphic chipset driver.

I am not going to reenter all that history. In brief, I have successfully installed the fglrx driver version 10.10 (the ATI Proprietary graphic chipset driver) on my laptop running the Linux Debian 5.0 Lenny Stable 64 bits.

Let's put it in perspective: I did that in 16 November 2010. In that day, the 10.10 version of fglrx is experimental, but not yet available in the experimental version of Debian. It may seem weired to install an experimental driver on the stable version of the OS, but I can say I had no choice. My graphic card was only supported starting from that version of fglrx. Yet I still stick to the reliable stable version of the Debian OS.

I am presenting this experience to the world because I find it useful for some of you, a bit like me. And I think it can help some others to install fglrx on their machines, if they have to. I made some bash scripts that automate the installation to some extent. But I want to show what they do first.

Hereafter are the general steps needed to install the fglrx driver. These are only meant to be understood, not to perfectly follow, because I will give some scripts that automate these actions just later.

1. Download the ATI Proprietary Linux Driver package. This got me the ati-driver-installer-10-10-x86.x86_64.run file.
http://support.amd.com/us/gpudownload/linux/Pages/radeon_linux.aspx

2. Remove all traces of ATI drivers using the package manager.

apt-get purge fglrx-driver fglrx-driver-dev fglrx-atieventsd fglrx-modules fglrx-kernel-src fglrx-control fglrx-glx fglrx-amdcccle fglrx-kernel fglrx-glx-ia32 fglrx-source

3. Extract ati-driver-installer-10-10-x86.x86_64.run into a directory fglrx-10-10.
"./ati-driver-installer-10-10-x86.x86_64.run" --extract "./fglrx-10-10"

4. Correct the internal structure of the fglrx installer package:
These steps are the fruit of hours of debugging efforts I did to figure out how to make the fglrx installer generate correct debian packages for me.

4.1. Create some directories.
The structure of the extracted installer files is not complete for a 64 bits system. So I corrected that.

4.2. Copy some files.
Some files needed to be copied to different locations inside the installer directory, in order to be installed in the right places when the packages are built.

4.3. Alter some files.
Some configuration files and scripts needed some changes to install the needed files and to make the correct links to the shared libraries when installed on the system.

5. Create the debian packages (.deb) from the extracted patched files.
./ati-installer.sh 10.10 --buildpkg Debian/lenny

6. Install the generated packages.

dpkg -i fglrx-driver*.deb
dpkg -i fglrx*.deb

7. Remove the open source ATI drivers available with Xorg.
apt-get purge xserver-xorg-video-ati

8. Patch some Linux source files.
This is required to solve a compatibility issue with fglrx and linux kernel API.
The issue is a change in the name of a function in the linux kernel, which is not taken into account by the developers of fglrx. The change I do does not really hurt the linux kernel itself, as all I am doing is adding a new function that is the exact copy of another one.

9. Build and install the fglrx kernel driver.
This will build the fglrx kernel module which enables the 3D acceleration of the graphic card.

10. Ensure that the fglrx kernel driver is loaded as a kernel module.
This is a final check to ensure the fglrx module has been installed and loaded into the kernel:
modprobe -l | grep fglrx

11. Configure the Xorg server to use fglrx driver.
First we have to stop gdm (GNOME Desktop Manager):
invoke-rc.d gdm stop

Then we generate the Xorg configuration that uses fglrx for X:
aticonfig --initial=dual-head

Finally we start gdm:
invoke-rc.d gdm start

12. Pin fglrx driver in the package manager to disallow any further automatic upgrades.
I don't actually provide the script needed to do this, as I easily do it using Synaptic Package Manager.

Here are the files needed. Just download this archive, extract it and put it in the same directory as the ati-driver-installer-10-10-x86.x86_64.run file. It will contain:

make_packages
install_driver
install_kernel_module
config_x
fglrx-driver.preinst
fglrx-driver.postrm

The scripts should be run as root from a real TTY (Alt-Ctrl-F2) as they don't expect X to be running all their lifetime. The run order is the following:

./make_packages 10 10
./install_driver
./install_kernel_module
./config_x

I am here for your comments or questions, don't hesitate.
I wish I added something to the community. Enjoy!

Murphy Laws

2010-09-04T08:49:00.000+01:00

Murphy Laws Site - The origin and laws of Murphy in one place.

http://www.murphys-laws.com/

5 Stages of Programmer Incompetence

2010-09-04T08:47:00.001+01:00

5 Stages of Programmer Incompetence

Hurl

2010-09-04T08:42:00.001+01:00

Perfect debugging solution for Web service programmers and consumers!
Experiment with the web service and see what comes out before you program it.

Hurl

Learn Lyx in 60min by K++

2010-08-03T00:26:00.002+01:00

Learn Lyx in 60min by K++
If the link is dead, ask me again for the file.

A study on Plug-in system concept

2010-07-19T21:48:00.004+01:00

A study on Plug-in system concept

Definitions

Plug-in: A program that forms a part of another program. It is designed to depend on the Plug-in server. It communicates with the server thru its Plug-in API.
Plug-in server: A program designed to manage plug-ins.
Plug-in API: A programming interface that connects the plug-in to the server.
Server environment: Global variables or constants defined by the server that are part of the plug-in API. They are common to all plug-ins.
Plug-in environment: Variables or constants defined by the server that are part of the plug-in API. They are specific to each plug-in.

Specifications

Plug-in

Self identification is optional unless required by the server.
After receiving unloading event, a plug-in must free all its associated resources and effectively unload from memory without leaving any executing code after it.
Can provide user interaction methods only if the server allows it.
It must be able to unload it self before the server does it. If this is wanted, the server must be notified about this.

Plug-in Server

Self identification.
Registration and removal handling.
Loading and unloading handling.
Providing the Plug-in API interface.
Maintaining compatibility in API interfaces.
Sending vital plug-in events: Loading and unloading.
If it defines environments variables, it must handle mutual access to them by the plug-ins.

Plug-in API

Interface can be of any type. Examples: DLL exported functions, DLL exported C++ classes, COM classes, IPC technologies (named pipes, sockets, RPC...), .NET interoperability, Java classes...
Different interfaces types may provide different capabilities but a basic level of capabilities must be shared by all theses.
Programming language independent, but may be binary dependent.
Error communicating methods are defined by the server.

Server environment

If a plug-in set them, other plug-ins must be notified.
A plug-in receiving environment-change-notification must be able to refuse the change or unload it self returning error to the server.
Each server environment entry must have a default value before loading the first plug-in.

General recommandations about performance

Plug-in API

Provide minimal interaction between the server and the plug-in.
Be simple and extendable. If advanced functionalities are required, they may be specified using special API entries or flags.
Provide a way to set environment entries in batch mode.
DLLs are very efficient as an interface so use them when you can, this implies using any DLL-based technology (DLL exported functions, DLL exported C++ classes, in-process COM classes).
IPC technologies provide other sets of options for the plug-in (executing code out-of-process, networking, operating system independency, possibility of using other programming languages…) but they are not recommended.

Networking implemented transparently in IPC technologies (socket, RPC using socket, COM Class Proxies) slows down significantly interaction performance and should be handled specifically and with care.
Executing out-of-process code requires explicit synchronization to be handled by the plug-in and adds much checking points before each interaction with the server and many features cannot be implemented using IPC communications.
Some older programming languages produce only executable modules (EXE) so they will oblige you to use IPC.
Operating system independency is very rarely required and it's very complex due to conversions needed before and after each single interaction and network management between both operating systems.

.NET and Java plug-ins are not recommended because loading a single module containing .NET or Java code would require the server to load a big part of the .NET framework or the Java Virtual Machine. This recommendation may be obsolete in future if the operating system is based on the .NET framework or the machine contains Java Microprocessor, in both cases loading the considered plug-in will be very efficient.
If a plug-in module must be registered somehow to run properly (VB6 modules, .NET assemblies, Java applets…) it's recommended to make an installation program to this plug-in.

Plug-in Server

Make plug-ins interaction in separate threads in order to maintain a responsive user interface and to take advantage of multithreading technologies.
If a server must make plug-in interaction in shared threads, it must provide a user interface showing that a plug-in interaction is in progress if such an interaction takes much time. This not recommended as it implies the ability of the user to "cancel" interaction which may be hard to implement or inefficient.
Use of structure versioning and flag masking is recommended for performance and compatibility

Plug-in

Avoid interaction with the server in loops.
If user interaction is designed, it should be separated from server interaction to maintain user interface responsiveness.
If multiple plug-ins are designed to the server, and the server supports multiple plug-ins in one module, then designing all the plug-ins in the same module is recommended in most cases.

General recommandations about security

Catch error cases from the plug-ins including fatal or unexpected errors. Example: SEH exceptions, hardware exceptions...
Using GUID to identify the plug-ins avoids collision between them.
If you have to choose between compiled, interpreted and script code, use compiled or interpreted when possible. Scripts present many security risks.

The trick to the Graceful Exit

2010-03-26T16:07:00.004+01:00

There's a trick to the Graceful Exit.
It begins with the vision to recognize when a job, a life stage, a relationship is over -- and to let go.
It means leaving what's over without denying its validity or its past importance in our lives.
It involves a sense of future, a belief that every exit line is an entry, that we are moving on, rather than out.
The trick of retiring well may be the trick of living well.
It's hard to recognize that life isn't a holding action, but a process.
It's hard to learn that we don't leave the best parts of ourselves behind, back in the dugout or the office.
We own what we learned back there.
The experiences and the growth are grafted onto our lives.
And when we exit, we can take ourselves along -- quite gracefully...

Grand Corps Malade - Mental

2010-03-22T15:32:00.002+01:00

S'il y a bien une idée qui rassemble, une pensée qu'est pas toute neuve
C'est que quel que soit ton parcours, tu rencontres de belles épreuves.

La vie c'est Mister Hyde, pas seulement Docteur Jekyll.
J'ai vu le film depuis longtemps, la vie n'est pas un long fleuve tranquille.

T'as qu'à voir les réactions d'un nouveau né à l'hôpital
S'il chiale si fort c'est qu'il comprend que souvent la vie va lui faire mal.

Y'a des rires, y'a des pleurs, y'a des bas, y'a des hauts.
Y'a des soleils et des orages et je te parle pas que de météo.

On vit dans un labyrinthe et y a des pièges à chaque virage
A nous de les esquiver et de pas calculer les mirages.

Mais le destin est un farceur, on peut tomber à chaque instant
Pour l'affronter, faut du cœur, et un mental de résistant.

J'ai des cicatrices plein la peau, et quelques unes dans mes souvenirs
Y'a des rescapés partout, j'suis un exemple, ça va sans dire
Ca doit se sentir, faut pas se mentir, la vie c'est .aussi la guérison
Après la foudre, prends toi en main et redessine ton horizon.

Y'a des tempêtes sans visage où on doit se battre contre le pire
Personne n'y échappe Rouda c'est pas toi qui vas me contredire
C'est l'ultime épreuve où tu affrontes la pire souffrance morale
Quand la peine rejoint l’impuissance pour la plus triste des chorales.

J'ai vu des drames à cœur ouvert j'ai vu des gens qui s'accrochent
Ce qui est bien avec le drame, c'est que tu le partages avec tes proches
Pour les miens il est peut-être l'heure de m'arrêter un court instant
Pour les remercier d'avoir du cœur et un mental de résistant.

La vie est aussi perverse, ce que tu désires elle l’a caché
Elle te le donnera pas tout cuit. il va falloir aller le chercher.

Du coup ce qu'on a, on le mérite, au hasard on a rien piqué
Et si t'as pas compris, va voir mes potes ils vont t'expliquer.

Jacky, tu m'as dit que l'ascenseur social était bloqué
Toi t'es allé chez Otis et le réparateur tu l'as braqué
Sans craquer, sans rémission, t'as affronté de vraies missions
Tu m'as montré qu'avoir du mental. c'est aussi avoir de l'ambition.

Y'a pas de chemin facile, Brahim t'as rien demandé à personne
T’as tout construit de tes mains et y'a pas que moi que tu impressionnes.

Toi Sami t'es notre moteur parce que tu sais depuis longtemps
Que pour que ca chémar il faut du cœur et un mental de résistant.

Je crois qu'on a tous une bonne étoile sauf que des fois elle est bien, planquée
Certaines même plus que d'autres, il faut aller les débusquer.

Parfois ça prend du temps quand tu fais trois fois le tour du ciel
Mais si tu cherches c'est que tu avances, à mon avis c'est l'essentiel.

Je fais partie de ceux qui pensent qu'y a pas de barrière infranchissable
Il faut y croire un peu, y'a bien des fleurs qui poussent dans le sable
Et' c’est quand tu te bats qu'il y a de belles victoires que tu peux arracher
Comme se relever avec une moelle épinière en papier mâché.

Je n'apprends rien à personne, tu es vivant tu sais ce que c'est.
Vivre c'est accepter la douleur, les échecs et les décès
Mais c'est aussi plein de bonheur, on va le trouver en insistant.
Et pour ça, faut du cœur et un mental de résistant.

Idir, La pluie

2010-03-03T20:56:00.003+01:00

Tant de pluie tout à coup sur nos fronts, sur nos champs, nos maisons.
Un déluge, ici l'orage en cette saison, quelle en est la raison?
Est-ce pour noyer tous nos parjures? ou laver nos blessures?
Est-ce pour des moissons, des terreaux plus fertils?
Est-ce pour les détruire?

Pourquoi cette pluie? Pourquoi? Est-ce un message, est-ce un cri du ciel?
J'ai froid mon pays, j'ai froid. As-tu perdu les rayons de ton soleil?
Pourquoi cette pluie? Pourquoi? Est-ce un bienfait? Est-ce pour nous punir?
J'ai froid mon pays, j'ai froid. Faut-il le feter ou bien le maudir?

J'ai cherché dans le livre qui sait, au creu de ses versets.
J'y ai lu: "Cherche les réponses à ta question. Cherche le trait d'union".
Une mendiante sur mon chemin, que fais-tu dans la rue?
Mes fils et mon mari sont partis un matin, aucun n'est revenu...

Pourquoi cette pluie? Pourquoi? Cette eau, ces nuages qui nous étonnent?
Elle dit cette pluie:
"Tu vois, ce sont des pleurs pour les yeux des Hommes.
C'est pour vous donner des larmes, depuis trop longtemps elles ont séché.
Les Hommes n'oublient pas les armes quand ils ne savent plus pleurer..."

Coule pluie, coule sur nos fronts...

Client/Server network communication using socket

2009-11-09T13:22:00.013+01:00

Task:

Make a server program which listens for connection from multiple clients. It receives data from the client and prints it on the standard output.
Make a client program which connects to the given server. It sends data to it.

Notes:

The clients are served in parallel, each connected client has a thread associated. There is no explicit limit on the number of connected clients.
The client is portable on Linux and Windows. It uses Winsock 2 API on Windows.
The server runs only on Linux because it uses the pthread API.
The connection protocol can use virtually any protocol (including IPv4 and IPv6).
The server uses the port 20200 by default. This can be changed in command line arguments.
The client uses port 20200 by default, and tries to connect to a server running on localhost. This can be changed in command line arguments.

Source package:
The source package contains the client and server sources. If you have Linux, extract its contents and run make. It will compile both of them. Once done, you will find a new directory called bin, inside which another called debug, inside which you find two executable files: the server nsa_server and the client nsa_client.
Here is a link for download: nsa.tar.gz

Wireshark screenshots:
Using wireshark, we watched the network activity between the client and server. We made a local connection (on the localhost).

The network activity is described as follows:

The first 3 frames indicate connection requests (SYN) from the client and the server along with connection acknowledgements.
The following 2 frames indicate that the client sent data to the server which acknowledged it. The data was "HELLO!!!".
The last 3 frames indicate connection closing by both client and server along with the respective acknowledgements.

Virus Programming

2009-10-23T10:52:00.004+01:00

The way through the mystery

What is a Virus?
Who makes a virus?
For what purpose?
How it gets done?
Would you do that yourself?

Task: Network monitoring using Wireshark

2009-10-21T13:19:00.007+01:00

Task:

In ENSI, connect to the wireless network available. Then open a site hosted in Tunisia.
Use Wireshark to monitor the network activity.
Filter the traffic of my own machine.
Generate a network traffic schematic and comment.

Work:
The site I'll be using is http://www.ensi.rnu.tn.

I connect to the network, install wireshark. Then open a shell, type:

to become a root.

I'm connected behind a proxy server whose IP address is 10.0.0.22 on port 80.
To clear ARP cache for the proxy server, I type:

arp -d 10.0.0.22

I need to clear firefox browser cache, to ensure the site I open is loaded entirely from network, not from local cache. So I type:

firefox &

then I use the menu command "Tools > Clear Recent History" and choose to clear everything. Then I close firefox.

I type:

ifconfig

This gives me the list of network interfaces available on my system. I find out that eth1 is the wireless network card. I note the MAC address of the interface as I'll need it later. My MAC address will be noted later myMAC.

I leave the console open and start wireshark as root to start capturing the traffic for the interface eth1.

I go back to the console and type:

firefox www.ensi.rnu.tn &

to access the site. Once the site is fully displayed, I close firefox, stop wireshark capturing and save the capture to a file.

In order to see only the traffic related to my computer, I filter the traffic by Ethernet II MAC address (either as a source address or as a destination address). The appropriate filter for doing that is:

eth.addr == myMAC

To further limit the traffic and see only the HTTP traffic, the wireshark filter becomes:

eth.addr == myMAC && http

There are several phases in the HTTP capture:

A HTTP GET request is sent to to the proxy server in order to initiate loading of site contents data.
The sent request is acknowledged by the server using a HTTP OK and specifies that the wanted information is a html page.
Then several HTTP GET requests are made to get the files embedded into the home page. Along with respective acknowledgements.
Finally a HTTP GET request is sent to get the site favourite icon, but in our case the site does not offer an icon. So the request is acknowledged by a HTTP 404 Not Found.

Please view the complete image for full details (the blog may display only a part of it, right-click on the image and select "view the image").

We change the filter to see only TCP and HTTP traffic, so it becomes:

eth.addr == myMAC && tcp

We note that the HTTP requests delimit the TCP requests, it means that HTTP requests start and end the data transfers.
These are some of the steps captured in wireshark:

In the beginning, the system establishes the connection with the proxy server, that's why it sends the server SYN TCP request. The server accepts the connection and acknowledges it by sending TCP SYN ACK frame. Then the system acknowledges the SYN ACK by sending the server a TCP ACK frame.
The system sends the HTTP GET request as mentioned earlier. The server acknowledges it first by sending the system a TCP ACK frame. Then it starts sending data to the system.
After each frame of data well received by the system, the system sends an acknowledgement to the server to allow it to send the next data frame.
After all data requested by the last HTTP GET request has been sent to the system, the proxy server sends the HTTP OK frame to indicate the end of data. As usual, this frame is acknowledged by the system by sending a TCP ACK frame.
In the very end of the capture, we observe the acknowledgements which involve TCP FIN frame. This ends the data transfer.

A simple simulator for the Intel 8086 16 bits microprocessor

2009-10-21T12:56:00.006+01:00

Currently, I8086 is a simulator application for 16 bits-based programs, this means that it tries to reproduce the same execution results for a given program as it might come out when executing the program on a real 8086 CPU.

It does not execute it in real-time, nor does it handle all the features of the processor.

I8086SIM is a set of 32-bits Windows-based programs written in C++ and inline x86 Assembly using Microsoft Visual Studio 2008 Service Pack 1 (Visual C++ version 9.0). It runs on Windows 2000 and later.

I8086SIM is command line driven. It can disassemble and simulate the execution of a 16 bits program in an isolated virtual machine enabling user inputs to the program and printing the outputs it produces. The program and the simulator work in two independent consoles, enabling consistent display and parallel monitoring.

I8086 was only built for simple demonstration purposes by Koutheir Attouchi in 2009.
Here are the files: