Gerd Riesselmann

Git Dubious Ownership on Remote Repository

2024-05-31T00:00:00+02:00

A couple of days ago I suddenly got the following error, when pulling from a remote Git repository using SSH:

fatal: detected dubious ownership in repository at '/some/repository'
To add an exception for this directory, call:

        git config --global --add safe.directory /some/repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I did as advised, and added /some/repository as safe directory. But the error remained.

Adding the repository to the list of safe directories did not work.

Turns out: You had to change the Git config on the server hosting the repository.

For this, a server administrator needs to execute sudo git config --system --add safe.directory '/some/directory' on the server.

If the server hosts more than one repository she may also execute sudo git config --system --add safe.directory '*'. This will mark all repositories on this server as safe.

Restart service if executable binary changes

2014-12-30T00:00:00+01:00

This is a simple bash script to restart a service if the executable binary has been changed, for example during deployment.

#!/bin/bash
SERVICE="{insert service_name}"
FILE="{insert /path/to/service/binary}"
FINGERPRINT="/root/watchers/$SERVICE.md5"

fingerprint_expected=$(/usr/bin/md5sum "$FILE")

if [ ! -e "$FINGERPRINT" ]
then
  echo "$fingerprint_expected" > "$FINGERPRINT"
fi

fingerprint_content=$(/bin/cat "${FINGERPRINT}")

if [ "$fingerprint_expected" != "$fingerprint_content" ];
then
  # Service has new binary
  echo "$fingerprint_expected" > "$FINGERPRINT"
  echo "Service $SERVICE has changed, trying to restart"
  /usr/sbin/service "$SERVICE" restart
fi

This is tested on a Debian system. I call it as a cron job. It is also available on GitHub.

Deploying an assembly jar file with sbt

2011-08-03T00:00:00+02:00

The assembly-sbt plugin for sbt 0.7, the (not so) simple build tool for scala, creates a nifty fat jar containing everything to run an application anywhere. You may want an additional task to automatically copy it to a server. Here’s how (Linux only, sorry).

The following code copies the assembly jar to a linux computer using ssh. So you need an ssh account there, and you will also need access via a private key. Read this tutorial on how to accomplish this. The key should be password-less, since it will not be possible to enter a password.

Add this to your Project class in project/build/Project.scala. I assume the assembly-sbt plugin is enabled.

Change the properties to match you settings.

import Process._

lazy val deployAction: Task = task {
  val account = "user@server"
  val src = assemblyOutputPath.absolutePath
  val target = "/dir/on/server/" + assemblyJarName
  val target_abs = account + ":" + target
  val keyfile = System.getProperty("user.home") + "/keyfile"

  log.info("Copying " + src + " to " + target_abs)
  List("scp", "-i", keyfile, src, target_abs) ! log

  // Optional. Create symbolic link to last version
  val symlink = "/path/to/symlink"
  log.info("Symlinking it to  " + symlink)
  val cmd = "ln -sf " + target + " " + symlink
  List("ssh", "-i", keyfile, account, cmd) ! log

  None
}

lazy val deploy = (deployAction && incrementVersion) dependsOn(assembly) describedAs("Builds assembly and moves it to server")

What the code is to take the assembly created by assembly-sbt and copies it to the server using scp. It then will create a symbolic link to the copied file, which therefore always points to the most recent version. After that it automatically increments the application’s version.

Run it using

sbt deploy

Authentication using Unfiltered for Scala

2011-07-11T00:00:00+02:00

Since it took me a while to figure this out (though it seems very obvious now), here’s how I do HTTP authentication in Unfiltered while preserving the authenticated user for later access.

class User(val name: String)

object User {
    def verify(name: String, password: String): Option[User] =
        if (
          name == "user" &&
          password == "pwd"
        ) {
          Some(new User(name))
        } else {
          None
        }
}

class AuthPlan extends Plan {
    val AuthFail =
        Unauthorized ~> WWWAuthenticate("""Basic realm="/"""")

    def intent = {
        case GET(Path("/na")) =>
            ResponseString("No Authentication needed")

        case req @ BasicAuth(u, p) =>
            User.verify(u, p) match {
                case Some(user) => req match {
                    case GET(Path("/a")) =>
                        ResponseString("Hello " + user.name)

                    case GET(...) =>
                        // Your other filters
                    case _ =>
                        NotFound
                }
                case None => AuthFail
            }

        case _ =>
            AuthFail
    }
}

Using /, ? and & in the path with Apache mod_rewrite and PHP

2011-06-06T00:00:00+02:00

I came upon a nasty problem the other day: The rel-tag microformat, poorly designed like so many of them, requires the URL to end on the tag name. Not a cleaned up, ANSI compatible version, no, a 1:1 representation. This on the one hand leads to some woes regarding encoding and non-ansi characters, think umlauts!, and on the other hand poses problems with caracters that usually are not allowed within an URL or have a special meaning. Think for example of a tag called “Q&A”.

Unfortunately, as we will see, simple escaping is not enough.

When it comes to routing requests to PHP code, a common approach is to use Apache’s mod_rewrite to redirect everything to a single file. The according code usually looks like this:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?path=$1 [L,QSA]

This code turns the current path into a query parameter named ‘path’.

However, the rewriting fails if the path contains either ? or &, even if they are url-escaped. For example

/q%26a

where %26 is the escaped ampersand,

with be turned into

index.php?path=q&a=

Note that the escaped ampersand has been turned into a real ampersand, thus dividing query parameters. We can verify this in PHP:

var_dump($_GET);

returns

path => q

a =>

Seems like Apache is ubersmart here.

There are two solutions to this problem:

1.) Use the B rewrite flag:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?path=$1 [L,QSA,B]

Using the ‘B’ prevents Apache from peeking into the path and from unescaping the ampersand. The path is now correctly turned into:

index.php?path=q%26a

And in PHP we get

path => q&a

The ‘B’ rule has been added in Apache 2. But for some reasons it doesn’t work reliable across distributions. While there are no problems on my local Ubuntu 10.4, there’s no effect on my webhoster’s Gentoo machine.

Luckily, there is a more generic solution

2.) Use double escaping

Escaping ?, & (and even /) twice allows safe usage of any of them. Escaping twice means to turn

? => %3F => %253F

& => %26 => %2526

/ => %2F => %252F

So, if we invoke

/q%2526a

we get

index.php?path=q%26a

And in PHP we get

path => q&a

Splitting large Axis generated files into separate classes

2010-08-08T00:00:00+02:00

I recently had to struggle with a SOAP webservice class generated by Apache Axis2. The class file was more than 13 MB in size, and contained more than 290.000 lines of code - much more than Eclipse, IntelliJ IDEA or NetBeans could handle. After some rather unsuccessful attempts to work around the problem, I finally decided to write a little tool to split the huge file into smaller pieces.

Apache Axis generates a stub class with all required parameter classes placed as nested classes within the stub class. Therefore I decided to extract each nested class into its own class file, turning it into a standard Java class. This additionally requires to rewrite all references to this class that use the stub class’s name.

Here’s the source. It’s also available on GitHub. It’s written in Scala, but should be easily transformable into any other language, since it just consists of a single method “main”.

import scala.io._
import java.io.FileWriter
import java.io.File

object Main {
  def main(args : Array[String]) : Unit = {
    // Directory where Axis generated files are, including slash at the end
    val in_dir = "C:/your/directory/"
    // Directory where to put extracted class files, including slash at the end
    val out_dir = in_dir + "extracted/"
    // Class name prefix fpr Stub And CallbackHandler classes
    val classname_prefix = "ChangeThis"
    // Class name of Axis generated stub and callback handler class
    val classname_stub = classname_prefix + "Stub"
    val classname_callback = classname_prefix + "CallbackHandler"
    // Namespace of Axis generated stub class
    val namespace = "your.name.space"

    var buffer_out : FileWriter = null
    // Everything that is not extracted is kept in stub file
    val buffer_main = new FileWriter(out_dir + classname_stub + ".java")
    val in = in_dir + classname_stub + ".java"
    val namespace_stub = namespace + "." + classname_stub + "."

    var nextmatch = 'c'
    for( l <- Source.fromPath(in).getLines("n")) {
      var line = l.replace(namespace_stub, "")
      nextmatch match {
        // Look for next static class declaration (This is the starting point of next snippet)
        case 'c' =>
          if (line.contains("public static class ")
               && !line.contains("Factory")
               && !line.contains("ExtensionMapper")) {
            var classname = line.replace("public static class", "")
            classname = classname.trim
            classname = classname.split(' ')(0)
            classname = classname.replace("{", "")
            buffer_out = new FileWriter(out_dir + classname + ".java")
            buffer_out.write("package " + namespace + ";nn")
            buffer_out.write(line.replace("public static", "public"))
            nextmatch = 'f'
          }
          else {
            // Write to stub class file
            buffer_main.write(line)
          }
        // Look for end of factory class. The curly bracket after this
        // is the end of class declaration
        case 'f' =>
          if (line.contains("//end of factory class")) nextmatch = 'b'
          // Just copy line
          buffer_out.write(line.replace("ExtensionMapper", classname_stub + ".ExtensionMapper"))
        // Look for closing curly bracket, ending extraction
        case 'b' =>
          buffer_out.write(line)
          if (line.contains("}")) {
            buffer_out.flush
            buffer_out.close
            // Look for next class
            nextmatch = 'c'
          }
      }
    }

    buffer_main.flush
    buffer_main.close

    // Change namespaces in CallbackHandler class
    val callback_file = new File(in_dir + classname_callback + ".java")
    if (callback_file.exists()) {
      val callback_out = new FileWriter(out_dir + classname_callback + ".java")
      for ( l <- Source.fromFile(callback_file).getLines("n"))
        callback_out.write(l.replace(namespace_stub, ""))

      callback_out.flush
      callback_out.close
    }

    println("Finished")
  }
}

To compile, place it a file, change the variables “in_dir”, “classname_prefix”, and “namespace”, and compile it using the Scala compiler or run it as a script. Here’s how to use the Scala compiler.

The code is more or less straight forward: Look for class declaration. When a class is found, copy stuff until the end of the class. The end of a class declaration is simply the next closing bracket, after a the comment stating the end of the factory class declaration. The Axis developers really should be thanked for this comment, since else it would be more complicated to determine the end of a class declaration. One should count curly brackets, I guess.

Creating Java Classes from WSDL file using Apache Axis 2

2010-08-06T00:00:00+02:00

Since I didn’t found this on the internet (but some rather outdated or complicated tutorials) here’s how to create Java class as wrapper for a given SOAP webservice.

Download most recent Apache Axis 2: http://ws.apache.org/axis2/download.cgi. I used version 1.5.1
Extract it to a directory of your choice
On the command line, change to subdirectory bin/ and run wsdl2java.bat -uri [URL of WSDL file] (on Windows) or wsdl2java.sh -uri [URL of WSDL file] on Linux.
The Java classes are created in the src/ subdirectory. Copy them into your project

To effectivly use the classes generated you must add a bunch of jars to your project dependiencies. These are found in the lib/ directory of the extracted Apache Axis distribution. You need ([version] notes a version number):

axis2-adb-[version].jar
axis2-kernel-[version].jar
axis2-transport-local-[version].jar
axis2-transport-http-[version].jar
axiom-api-[version].jar
axiom-dom-[version].jar
axiom-impl-[version].jar
commons-codec-[version].jar
commons-logging-[version].jar
commons-httpclient-[version].jar
httpcore-[version].jar
mail-[version].jar
neethi-[version].jar
wsdl4j-[version].jar
XmlSchema-[version].jar

Apache Axis generated a class class [Name of Webservice]Stub, e.g. for a webservice TimeOfDay it would be TimeOfDayStub. This class has all the webservice’s methods and nested classes for the method’s arguments. Everything is relatively straight forward and easy to use.

That’s it.

There is an Apache Axis Eclipse plugin for converting WSDL into Java classes, but the download URL seems broken. I also tried to use Eclipse’s WSD toolset, but this unnecessarily requires a local web server to be set up and running.

Edit: Corrected wsdl2java parameter from -url to -uri

PHP und UTF-8 - eine Anleitung, Exkurs: Wenn die DB durcheinander gerät

2010-06-09T00:00:00+02:00

Manchmal kommt es vor, dass der Import von Daten in die DB fehlschlägt, und man trotz korrektem UTF-8 wieder mit verhunzten Umlauten zu tun hat. In diesem Fall kann die CONVERT-Funktion von MySQL in Verbindung mit BLOB-Spalten hilfreich sein.

Hier ist ein entsprechender Test-Case:

CREATE TABLE test (
  id integer unsigned NOT NULL auto_increment,
  content varchar(45) NOT NULL,
  PRIMARY KEY  (`id`)
) DEFAULT CHARSET=utf8;

INSERT INTO test (content) VALUES ('KÃ¶ln');

Dies erzeugt eine Tabelle, in dessen UTF-8-Spalte “content” eine nach UTF-8 konvertierte ANSI-Version von “Köln” steht:

SELECT * FROM test;

id  |   content
--------------------
1   |  KÃ¶ln

Diese Spalte soll nun so konvertiert werden, sodass dort korrekt “Köln” steht.

Dazu wird nun zunächst einmal eine temporäre Spalte erzeugt, die den konvertierten Inhalt aufnimmt. Diese Spalte ist zunächst Latin1!

ALTER TABLE test
  ADD COLUMN c TEXT CHARACTER SET latin1;

Nun konvertieren wir den verhunzten UTF-8-Inhalt zunächst einmal nach Latin1:

UPDATE test SET c = CONVERT(content USING latin1);

SELECT * FROM test;

id  |   content  |  c
------------------------------
1   |  KÃ¶ln     |  KÃ¶ln

Die Spalte “c” enthält nun im Prinzip unser UTF-8 Ergebnis, wenn nur die einzelnen Bytes betrachtet werden. Es müsste nur der Zeichensatz geändert werden, ohne dabei eine Konvertierung durchzuführen.

Dies geht über den Umweg eines BLOB-Feldes:

ALTER TABLE test MODIFY COLUMN c BLOB;
ALTER TABLE test MODIFY COLUMN c TEXT CHARACTER SET utf8;

SELECT * FROM test;

id  |   content  |  c
------------------------------
1   |  KÃ¶ln     |  Köln

Et voilá!

PHP und UTF-8 - eine Anleitung, Teil 3: PHP String-Funktionen

2010-02-18T00:00:00+01:00

Es ist nun schon eine Weile her seit der letzten Folge, Zeit die unter anderem in den Gyro-PHP Application Framework geflossen ist. Dafür soll es in der Serie über PHP und UTF-8 nun endlich um PHP selber gehen. Dabei wird in einem ersten Schritt über die String-Funktionen von PHP zu reden sein, während die Frage der Regular Expressions auf eine weitere Folge vertagt ist.

Der grundlegende Unterschied zwischen Zeichensätzen wie Latin-1 aka ISO-8859-1 und UTF-8 ist, dass erstere pro Zeichen genau ein Byte brauchen, während ein Zeichen in UTF-8 beliebig viele Byte lang sein kann. Generell gilt die Regel, dass in UTF-8 die ASCII-Zeichen (das sind die Zeichen 1-128) ein Byte lang sind, die nächsten 1664 Unicode Zeichen (wozu auch das komplette Latin1 gehört) zwei Byte, und alle weiteren Zeichen (Kyrillisch, Indisch, Arabisch, etc.) drei Bytes.

Folgende Tabelle soll das illustrieren. Dargestellt werden die hexadezimalen Bytecodes eines Zeichen in Latin1 und UTF-8:

Zeichen	Code Latin1	Code UTF-8	Länge UTF-8
A	0x41	0x41	1
Ä	0xC4	0xC384	2
±	0xB1	0xC2B1	2

Zu den drei Byte langen Zeichen gibt es naturgemäß in Latin-1 keine Entsprechungen.

Dass ein Zeichen in UTF-8 keine feste Länge mehr hat, stellt nun viele String-Funktionen in PHP vor ein großes Problem: Denn diese setzen vielfach voraus, dass ein Byte immer ein Zeichen ist. Augenfällig ist dies bei der strlen() Funktion.

<?php
print strlen('Ä'); // ergibt 2, wenn die Datei in UTF-8 ist
?>

Weitere Funktionen, die mit UTF-8 Probleme bekommen sind unter anderem strtoupper() und strtolower() sowie strpos(). Erstere würden beispielsweise das große Ä nicht erkennen, da es aus zwei Bytes besteht. strpos() leidet unter den gleichen Problemen wie strlen(), denn es zählt zur Positionsermittlung einfach Bytes.

Zum Glück ist die Abhilfe hier ziemlich einfach. Denn für jede Funktion, die mit UTF-8 nicht funktioniert, existiert ein Pendant in der PHP-Erweiterung Multibyte String. Die jeweilige Ersatz-Funktion ist namensgleich mit der originalen PHP-Funktion, hat aber ein “mb_” vorangestellt. So ist etwa mb_strtolower() die Mutibyte-Variante zu strtolower().

Die einzige Ausnahme ist die Funktion lcfirst(), die erst mit PHP 5.3 eingeführt wurde. Diese ist nicht UTF-8-fähig, und es gibt keine Mutibyte-Ersetzung.

Alle Multibyte-Funktionen, die PHP-Funktionen ersetzen, sind um einen Parameter erweitert, der die Zeichenkodierung aufnimmt.

<?php
$l = mb_strtolower('Äh...', 'UTF-8');
?>

Dieser Parameter ist optional, der Default-Wert kann über den Parameter mbstring.internal_encoding in der php.ini oder über die Funktion mb_internal_encoding() gesetzt werden.

<?php
mb_internal_encoding('UTF-8');
$l = mb_strtolower('Äh...');
?>

Es ist möglich, durch eine Einstellung in der php.ini alle relevanten PHP-String-Funktionen durch ihre Multibyte-Äquivalente zu ersetzten. Hier die Dokumentation dazu. In diesem Fall ist mb_internal_encoding() unbedingt aufzurufen, es sei denn, die Eigenschaft mbstring.internal_encoding ist in der php.ini bereits entsprechend gesetzt.

Das Überladen der originalen PHP-Funktionen sollte aber nur der letzte Ausweg sein, um bestehenden Code UTF-8 fähig zu machen, dessen Portierung zu umfangreich ist. Ansonsten sollten die mb-Funktionen immer vorgezogen werden, da damit auch für Dritte bereits im Code zu sehen ist, dass man sich Gedanken über UTF-8 gemacht hat.

Die Multibyte-Erweiterung bringt auch Funktionen für das Arbeiten mit regulären Ausdrücken mit. Allerdings nur für die Gruppe der ereg-Funktionen, die ab PHP 5.3 als “deprecated”, also veraltet, markiert sind und nicht mehr verwendet werden sollten. Die preg-Varianten der Regular Expressions sind ebenfalls nicht ohne weiteres UTF-8-fähig, dafür existieren aber keine Überladungen. Dies wird Thema der nächsten Folge sein.

Neben den klassischen String-Funktionen ist eine zweite Gruppe von Funktionen zu diskutieren, die für den Gebrauch mit UTF-8 angepasst werden müssen. Dies sind die Konvertierungsfunktionen, die Zeichen in ihre entsprechenden HTML-Entities umwandeln. Folgende Funktionen sind betroffen:

htmlentities()
html_entity_decode()
htmlspecialchars()

Jede dieser Funktionen aktzeptiert, das wird gerne vergessen, einen dritten Parameter, der den Zeichensatz bestimmt. Mit UTF-8 müssen diese also so aufgerufen werden:

<?php
$safe = htmlentities($unsafe, ENT_QUOTES, 'UTF-8');
$unsafe = html_entity_decode($safe, ENT_QUOTES, 'UTF-8');

$safe = htmlspecialchars($unsafe, ENT_QUOTES, 'UTF-8');
// htmlspecialchars_decode() funktioniert immer, denn <, >, & sind immer ASCII
$unsafe = htmlspecialchars_decode($unsafe, ENT_QUOTES);
?>

Leider führt das unter PHP 4.3 zu einem Fehler, auch wenn UTF-8 als Parameter laut Dokumentation unterstützt werden sollte.

Bei der Portierung existierenden Codes von Latin-1 zu UTF-8 ist es am einfachsten, für jede der drei Funktionen eine eigene Funktion zu schreiben, etwa indem man ein ‘utf8_’ davor schreibt:

<?php
/**
 * UTF-8 aware version of htmlentities()
 */
function utf8_htmlentities($text, $quote = ENT_COMPAT, $charset = 'UTF-8', $double_encode = TRUE) {
  return htmlentities($text, $quote, $charset, $double_encode);
}

/**
 * UTF-8 aware version of html_entity_decode()
 */
function utf8_html_entity_decode($text, $quote = ENT_COMPAT, $charset = 'UTF-8') {
  return html_entity_decode($text, $quote, $charset);
}

/**
 * UTF-8 aware version of htmlspecialchars()
 */
function utf8_htmlspecialchars($text, $quote = ENT_COMPAT, $charset = 'UTF-8', $double_encode = TRUE) {
  return htmlspecialchars($text, $quote, $charset, $double_encode);
}
?>

Jetzt können die alten Aufrufe einfach per Suchen und Ersetzen durch ihre UTF-8-Varianten ersetzt werden.

An dieser Stelle sei gesagt, dass es generell eine gute Idee ist, Funktionen mit vielen konstanten Parametern, wie etwa hier der Quote-Style und der Charset, von vornherein in eine eigene Funktion zu wickeln. Dies macht es einfach, das Standardverhalten später zu ändern.

PHP und UTF-8 - eine Anleitung, Teil 2: Webseiten als UTF-8 ausliefern

2007-11-23T00:00:00+01:00

Nachdem nun die MySQL-Datenbankverbindung reibungslos mit UTF-8 funktioniert und bevor wir uns im weiteren Verlauf der Serie über PHP und UTF-8 wirklich PHP selber zuwenden, sollen noch ein paar Worte über das korrekte ausliefern einer UTF-8 kodieren Webseite verloren werden. Das Problem, das hier angegangen werden soll ist: Wie erkennt der Browser eigentlich, dass eine Webseite in UTF-8 ist?

Die Antwort ist

Am Content-Type-Feld im HTTP-Header
Nur wenn der HTTP-Header fehlt: Am Content-Type Meta-Tag im HTML-Code

Betrachten wir zunächst den HTTP-Header. HTTP kennt jeder von der Adresseingabe im Browser, und dahinter verbirgt sich ein Protokoll, das regelt, wie Daten vom Server zum Client (in diesem Fall der Browser) transportiert werden. Dazu werden vor dem eigentlichen Inhalt so genannte Header gepackt, die Informationen über den folgenden Inhalt wie auch etwaige Fähigkeiten des Clients liefern. Hier Auszüge des Headers für den Aufruf der Startseite von gerd-riesselmann.net:

HTTP/1.x 200 OK
Date: Thu, 22 Nov 2007 18:20:20 GMT
Last-Modified: Thu, 22 Nov 2007 18:20:20 GMT
Content-Type: text/html; charset=utf-8

Interessant ist für uns lediglich das Feld Content-Type. Dieses legt fest, das im Folgenden Text– und zwar HTML – folgt, der in UTF-8 kodiert ist. Dieses Feld – und erst mal nur dieses! – bringt den Browser dazu, die Seite als UTF-8 anzuzeigen. Wenn hier andere Sachen stehen etwa

Content-Type: text/html; charset=iso-8859-1

wird die Seite entsprechend als Latin1 angezeigt. Dagegen kann man dann auch nichts machen.

Deshalb muss zunächst dafür gesorgt werden, dass der Content-Type-Header richtig gesetzt wird. Dies kann auf mehrere Arten geschehen:

Aufseiten des Webservers
Durch PHP

Um etwa im Apache-Webserver UTF-8 auszuliefern, benötigen wir die Direktive AddDefaultCharset . Diese kann bei der Definition des Virtual Host in der Apache-Config gesetzt oder mittels einer .htaccess-Datei überschrieben werden. Die Syntax ist einfach:

AddDefaultCharset utf-8

Sollte das Setzen oder Überschreiben mittels .htaccess nicht möglich sein, etwa weil der Provider entsprechende Restriktionen hat, kann der Header auch mittels PHP gesetzt werden:

<?php
header("Content-Type: text/html; charset=utf-8");
?>

Um einen Header mittels PHP senden zu können, darf allerdings noch keine Ausgabe erfolgt sein, also macht man das am besten noch während des Programmsstarts.

Was passiert, wenn ein Content-Type-Eintrag im Header fehlt, etwa weil die Datei von der lokalen Festplatte geöffnet wurde? In diesem Fall greift der Browser auf den HTML-Code zurück, und hier auf einen Meta-Tag, das den Content-Type definiert:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
...
</head>
<body>
...
</body>
</html>

Das Attribut “http-equiv” bedeutet eigentlich, dass der Webserver seinen eigenen Content-Type-Eintrag im Header durch den Wert des Meta-Tags ersetzten soll. Aus naheliegenden Gründen wird das allerdings von keinem Webserver umgesetzt: Jedes Schnipsel HTML zu interpretieren würde einfach zu viel Last bedeuten. Daher greift nur der Browser hierauf zurück, und auch nur, wenn er keinen entsprechenden Header-Eintrag vorfindet.

Erwähnt sei noch, dass für XHML-Seiten das Encoding auch in der XML-Deklaration gesetzt werden kann:

<?xml version="1.0" encoding="utf-8"?>

Allerdings kann der Internet Explorer 6 mit solchen XML-Deklarationen nicht umgehen, weshalb sie für gewöhnlich weggelassen werden.

PHP und UTF-8 - eine Anleitung, Teil 1: MySQL

2007-11-22T00:00:00+01:00

Am Anfang der Serie über PHP und UTF-8 soll zunächst ein Abstecher zu MySQL stehen, schließlich ziehen die meisten PHP-Anwendungen ihre Daten aus einer MySQL-Datenbank. Und anders als man vielleicht annehmen würde, ist die Verwendung von UTF-8 mit MySQL alles andere als problemfrei - insbesondere reicht es nicht aus, Tabellen einfach als UTF-8 zu deklarieren!

Das Gemeine an einer fehlerhaften Verwendung von UTF-8 im Zusammenspiel von PHP und MySQL ist, das diese nicht sofort auffällt. In der Tat gibt es nur einen wirklichen Testfall für eine richtige UTF-8-Integration, und das ist die Erkennung mit dem LIKE-Operator und dem Buchstaben „a“.

Hier ist ein Testcase. Der Testcase wurde auf einem Linux-System ausgeführt, das UTF-8 als Systemzeichensatz benutzt. Auf System mit anderem Systemzeichensatz kann das Verhalten reproduziert werden, indem die SQL-Statements in eine Textdatei geschrieben werden, die als UTF-8 gespeichert und dann im MySQL-Client ausgeführt wird.

Starten Sie den MySQL-Client aus der Kommandozeile (nein – PHPMyAdmin, MySQL Query Browser oder andere Tools zählen nicht!):

mysql -uroot -p[rootpasswort] --default-character-set=LATIN1</code>

und geben Sie folgende Zeilen ein:

create database utf8test default character set 'UTF8';
use utf8test
create table sorttest (value varchar(15)) default character set 'UTF8';
insert into sorttest (value) values ('aaa');
insert into sorttest (value) values ('üüü');
select * from sorttest where value like 'a%';</code>

Das erwartete Ergebnis ist natürlich eine Zeile, nämlich „aaa“ – wir erhalten jedoch zwei: „aaa“ und „üüü“!

+--------+
| value  |
+--------+
| aaa    |
| üüü    |
+--------+

Wie kann das angehen?

Der Grund liegt natürlich im Parameter –default-chararacter-set=LATIN1. Machen wir mal die Gegenprobe, indem wir explizit UTF-8 als Standard setzen. Dazu öffnen wir wieder die MySQL Kommandozeile:

mysql -uroot -p[rootpasswort] --default-character-set=UTF8</code>

und geben zunächst folgendes ein:

use utf8test
select * from sorttest where value like 'a%';

Wir bekommen wieder zwei Ergebnisse, diesmal aber sehen wir etwas anderes:

+--------------+
| value        |
+--------------+
| aaa          |
| Ã¼Ã¼Ã¼       |
+--------------+

Hier wird zunächst klar, warum wir zwei Ergebnisse für die Abfrage „Beginnt mit A“ bekommen: „Ã“ wird auch unter „a“ eingeordnet. Gleichzeitig sehen wir, was wirklich in die Tabelle geschrieben wurde: Der UTF-8-Code für „Ã“ und der UTF-8-Code für „¼“, statt wie erwartet die ANSI-Zeichen-Kombination „Ã¼“, die den UTF-8-Code für „ü“ bildet.

Wir haben damit die wunderbare Welt der MySQL-Zeichensatzkonvertierung betreten. Machen wir aber zunächst weiter mit der Gegenprobe:

create table sorttest2 (value varchar(15)) default character set 'UTF8';
insert into sorttest2 (value) values ('aaa');
insert into sorttest2 (value) values ('üüü');
select * from sorttest2 where value like 'a%';</code>

Wir bekommen das korrekte Ergebnis:

+-------+
| value |
+-------+
| aaa   |
+-------+

Wir haben in obigen Beispielen die Zeichenkodierung des MySQL-Clients gesetzt. Lassen wir den Parameter –default-character-set“ weg, wird die Standard-Kodierung genutzt. Sofern diese nicht manuell angepasst worden ist (was selten passiert) ist das Latin1 mit der Sortierung „Schwedisch“!

Dröseln wir jetzt mal auf, was da genau passiert ist:

Wir sind auf einem Linux-System mit UTF-8 als Zeichensatz. Die Eingabe „üüü“ wird daher in die ANSI-Zeichen „Ã¼Ã¼Ã¼“ umgesetzt.
Der MySQL-Client erwartet Latin1, liest also nicht „üüü“, sondern eben „Ã¼Ã¼Ã¼“.
Und dies schickt er auch an den Server, und zwar mit der Anmerkung, es handele sich hier um Daten im Format Latin1.
Der Server wiederum weiß, dass die Tabelle UTF-8 benutzt und konvertiert entsprechend die ankommenden Daten von Latin1 nach UTF-8. Er speichert also nicht ein UTF-8-"ü", sondern UTF-8-„Ã¼Ã¼Ã¼“
Wir fragen die Tabelle ab und verlangen dabei das Format Latin1.
Der Server gibt uns die Daten zurück, konvertiert aber vorher von UTF-8 nach Latin1, weil der Client das so wollte.
Der Client erhält „Ã¼Ã¼Ã¼“ als ANSI und druckt das aus.
„Ã¼Ã¼Ã¼“ werden auf dem Bildschirm als „üüü“ angezeigt.

Offensichtlich gibt es hier zwei Knackpunkte:

Die Zeichenkodierung der Eingabe
Die Konvertierungen auf dem Weg vom Client zum Server

Wenn wir eine PHP-Seite betreiben, die ihre Daten als UTF-8 ausgibt, werden auch die Eingaben aller Forms als UTF-8 vorliegen. Sobald wir diese in die Datenbank speichern wollen haben wir effektiv die gleiche Situation wie im obigen Testcase, da MySQL mit hoher Wahrscheinlichkeit „Latin1/Schwedisch“ erwartet.

Wir müssen also MySQL aus PHP heraus irgendwie mitteilen, dass UTF-8-Daten kommen, und das geht über das „SET NAMES“- bzw. das „SET CHARSET“-Kommando. Direkt nach der Verbindung zur Datenbank senden wir dazu folgendes SQL-Statement:

set names 'utf8';

Dies entspricht dem Parameter –default-character-set=“UTF8“ beim MySQL-Client.

Das wars.

Naja, nicht ganz. Wir können natürlich an der MySQL-Konfiguration herumschrauben und dort UTF-8 als Standard setzen. Hier der entsprechende Auszug aus meiner MySQL-Konfiguration :

[mysql]
default-character-set=utf8

[mysqld]
collation_server=utf8_unicode_ci
character_set_server=utf8

Allerdings kann das zu unangenehmen Überraschungen führen, wenn das System migriert wird. Und wenn erst einmal richtige UTF-8 und ANSI-UTF-8-Mischmasch-Daten in einer Tabelle stehen ist Spaß garantiert.

Des Weiteren sollte man sich angewöhnen, alle MySQL-Clients mit dem Systemzeichensatz als Zeichensatz zu starten, im Fall eines UTF-8-basierten Linux also so:

mysqldump --default-character-set="UTF8" -u... -p...
mysql --default-character-set="UTF8" -u... -p...
mysqlimport --default-character-set="UTF8" -u... -p...

PHP und UTF-8 – eine Anleitung, Einleitung

2007-11-22T00:00:00+01:00

Es ist kein Geheimnis, dass PHP und Multibyte-Zeichensätze wie etwa UTF-8 nicht gut zusammen gehen. Dies hat drei Gründer:

Die meisten PHP-Funktionen setzen wie selbstverständlich ANSI-Strings voraus, etwa die meisten String-Funktionen
Bei PHP-Funktionen, die Multibyte-Zeichensätze verarbeiten können, wie etwa htmlentities(), werden diese nicht benutzt.
Und last not least ist der korrekte Einsatz von MySQL und UTF-8 auch nicht einfach.

Ich werden diesen drei Problemfeldern nachspüren.

Teil 1: UTF-8 und MySQL

Teil 2: Webseiten als UTF-8 ausliefern

Teil 3: PHP String Funktionen

Exkurs: Exkurs: Falsche Daten in der DB fixen

Apache Xerces' weird error message

2007-04-04T00:00:00+02:00

For one of my projects I’m using Apache’s Xerces C++ XML Parser and run into this strange error:

The primary document entity could not be opened.

Took me a while to figure out that Xerces simply wants to say “File not found” here, because I was was calling parse() with the data to be parsed, while Xerces expects a file name here. Silly me.

However: Here’s how to parse an XML string using Xerces SAX parser. You need to use a MemBufInputSource:

#define XNS XERCES_CPP_NAMESPACE_QUALIFIER 
std::string xmldoc("<first><second>Some Text</second></first>");
//

//  Create a SAX parser object.

//

XNS SAX2XMLReader* parser = 
	XNS XMLReaderFactory::createXMLReader();
//

//  Create the handler object and install it

//

YourContentHandler handler();
parser->setContentHandler(&handler);
parser->setErrorHandler(&handler);
XNS MemBufInputSource source(
	(const XMLByte*)xmldoc.c_str(),
	xmldoc.length(),
	"a name for the input source",
	false
);
parser->parse(source);

When Internet Explorer displays a white page...

2007-03-27T00:00:00+02:00

.. this may be because of a wrong script tag. Actually, both Internet Explorer 6 and 7 will fail, if you use the script tag like this:

<script type="text/javascript" src="blabla.js" />

IE instead requires an explicit closing tag, like this:

<script type="text/javascript" src="blabla.js"></script>

PS: This is primary a note to myself, since I get trapped by this over and over again…

UTF-8, Latin1 aka ISO-8859-1 und das Euro-Zeichen

2006-11-18T00:00:00+01:00

Um mal gleich mit der Tür ins Haus zu fallen: Es gibt kein Euro-Zeichen im ISO-8859-1-Zeichensatz (aka Latin1) - und folglich sind alle Versuche, einen Text mit dem niedlichen Kniddel-E von UTF-8 oder jedem anderen Zeichen-Encoding nach Latin1 zu konvertieren zum Scheitern verurteilt. Mit etwas Glück kommt dabei ein Fragezeichen oder immerhin noch Gekröse raus. Korinthenkacker wie iconv dagegen stellen einfach ganz die Arbeit ein und man endet in einem Fehler.

Latin1 soll also kein Euro-Zeichen enthalten, hmm? Warum kann ich dann das “€” in Webseiten sehen, die ISO-8859-1 kodiert sind?

Tja… Sagen wir mal so: Es ist heute ganz einfach, durch ganz Europa zu reisen und seine Euros fröhlich überall auszugeben. Das heißt aber nicht, dass ein Grenzer einen nicht trotzdem stundenlang mit Formalitäten an der Grenze festhalten könnte.

Das ist bei Computern nicht groß anders: Es gibt freundliche Programme, die versuchen, das Leben angenehm zu machen, etwa indem sie Konventionen gelten lassen. Eine solche Konvention sagt, das man das Zeichen 128 im Latin1-Zeichensatz durch das Euro-Symbol ersetzt hat.

“Man” ist in diesem Fall eine große Koalition aus Software- und Hardware-Herstellern. Schließlich musste bei der Einführung des Euros eine einfache und billige Lösung her. Die Vorteile liegen auf der Hand: die Programme selber müssen nicht geändert werden, es braucht lediglich eine neue Schrift. Die Nachteile der Lösung kennt jeder, der beim Drucken auf älteren Druckern auf die installierten Schriften des Druckers zurück greifen muss, die das Euro-Zeichen leider nicht kennen.

Auf der anderen Seite gibt es Programme, die penibel auf die Einhaltung der Regeln pochen: bürokratisch, humorlos. Wie zum Beispiel die Programme, die Zeichenkodierungen konvertieren.

Diese argumentieren, ISO-8859-1 enthalte ganz offensichtlich kein Euro-Zeichen (bei der Festlegung von ISO-8859-1 gab es noch keinen Euro) und gleich dreimal nicht an Position 128. Stattdessen gäbe es doch die Kodierung ISO-8859-15, diese definiere ein Euro-Symbol und sei doch auch bereits seit 1999 gültig.

Und das ist natürlich nicht verkehrt. ISO-8559-15 enthält einen Euro, und zwar an Position 164. Damit sind ISO-8859-15 und ISO-8859-1 (Euro-tauglich) euro-mäßig wiederum nicht kompatibel. Das wäre ja auch zu einfach, erklärt sich aber vielleicht daraus, dass bei Latin1 an Position 164 noch dieses Zeichen steht: ¤. Das ist das internationale Währungszeichen - süß, nicht wahr? Besser bekannt ist dieses Zeichen freilich als Tabellenzellen-Ende-Zeichen aus Microsoft Word und anderen Textverarbeitungen. Genau so gut hätte man vorschlagen können, das Symbol ¶ durch den Euro zu ersetzen.

ISO-8859-15 jedenfalls ist ein lustiger Geselle, von dem man noch nicht mal weiß, ob er nun Latin0 oder Latin9 heißt. Man weiß jedoch, dass er eigentlich Latin1 hatte ersetzen sollen. Daraus ist allerdings nichts geworden, wenn wir uns zum Beispiel PHP, QT oder andere Programmiersprachen und -frameworks ansehen. Latin1 ist immer noch omnipräsent.

Wer sich aber Ärger ersparen will, der greift fortan auf ISO-8859-15 zurück.

Aus all dem folgt also, dass Code wie der folgende - der einen UTF-8 Text annimmt und eine Latin1-Mail verschickt - in die Katastrophe führen, wenn der Text ein Euro-Zeichen enthält:

<?php
function send_mail($to, $subject, $utf8_message) {
  //falsch!
  $latin_message = @iconv('UTF-8', 'ISO-8859-1', $utf8_message);
  // auch falsch!
  $latin_message = @utf_decode($utf_message);
  mail($to, $subject, $latin_message);
}

Hier ist die richtige Version. Da wir latin0-oder-9 verwenden, müssen wir das beim Verschicken der Mail irgendwie mitteilen, also auch einen entsprechenden Header dazupacken:

<?php
function send_mail($to, $subject, $utf8_message) {
  $latin_message = @iconv('UTF-8', 'ISO-8859-15', $utf8_message);
  $header = 'Content-Type: text/plain; charset=ISO-8859-15';
  mail($to, $subject, $latin_message, $header);
}

The Holy Grail CSS Layout Fix for IE7

2006-11-08T00:00:00+01:00

In Search of The Holy Grail describes an easy way to create a three-column, source-ordered, table-less layout with a fluid column in the middle. It is used by the Comic Marktplatz, for example. However, there is an annoying bug with the shiny new Internet Explorer 7. IE7 has a problem in understanding the left column rule of

{ margin-left: -100%; }

Instead of considering the width of the surrounding div container, IE7 inserts the width of the body, which leads to the left column being moved out of sight to the far left. Here’s how to fix it.

I experimented a little with overflow, position, and other attributes without any success. However, since IE supports JavaScript expressions in CSS, this can be used to apply a fix (The fix is a fix to this fix, actually):

#left {
  width: 200px;             /* LC width */
  margin-left: -100%;
  left: 150px;              /* RC width for IE6 */
}
#container > #left {
  left: -200px;             /* -LC width for others */
  margin-left: expression(document.all.center.offsetWidth * -1); /* Fix for IE7 */
}

In the case of the padded version, the left and right padding of the left and the left padding of the center column must be added to the margin like this:

#container > #left {
   left: -200px; /* -LC width for others */
   margin-left: expression(
   document.all.center.offsetWidth * -1 +
   parseFloat(document.all.center.currentStyle.paddingLeft) +
   parseFloat(document.all.left.currentStyle.paddingLeft) +
   parseFloat(document.all.left.currentStyle.paddingRight)
   ); /* Fix for IE7 */
}

“left” in document.all.left is the id of the left column.

Attention: Note that “center” within the expression document.all.center.offsetWidth refers to the column with the id “center”, as it was named in the original version. If you named the middle column differently, you must change this, too.

And again: Is C++ really faster?

2006-11-04T00:00:00+01:00

I was pointing to articles about if low level optimization really makes sense and if C++ is really faster than higher level languages several times. But of course this will be a never ending debate, and it still continues. This time C and C++ should be the best languages for higher level mathematics problems. Unfortunately this is not the case.

The author argues that

C and C++ just aren't productive languages. It takes 2-4 times as long to implement something in these languages than in a very high level language like Ruby. The resulting code is harder to read, harder to maintain, and harder to port.

I totally agree with this, and like the author I think this is the reason why C++ looses popularity.

I disagree however when the author opposes to high level language features like a garbage collection. He argues that this will have impact on speed, which is the last outstanding C and C++ have to offer:

C and C++ are never going to disappear. Why? Because there are classes of programming problems that are still and will always be CPU bound and there is still no language as fast as C or C++ for these problems. I highly doubt that there ever will be. I'm talking about things like: scientific number crunching, game/simulation physics, raytracing, real-time 3d graphics, audio processing, codec implementation, high-speed network packet routing, evolutionary computation (my personal favorite :), and of course implementing all these high-level languages' runtimes. There are also problems like OS and hardware driver implementation where you need something "close to the metal" that can interact closely with and even embed assembly language. C is basically shorthand for assembler, which is why it's the preferred language for that kind of thing.

While the second part of the argument may be true (I’m not that into C and Assembler since I sold my C64 back in 1985), this would be a rather - well - boring niche for the powerful language C++ is. For the first part, this is plain wrong, as it was already stated in the comments.

A more detailed reply can be found at Good Math, Bad Math - and it is the same as for low level optimization: Compiles do a much better job than people can do. If they are provided with the necessary information, which due to their pointer based nature is a problem in C and C++:

So for modern systems, writing an efficient program is sort of a partnership. The human needs to careful choose algorithms - the machine can't possibly do that. And the machine needs to carefully compute instruction ordering, pipeline constraints, memory fetch delays, etc. The two together can build really fast systems. But the two parts aren't independent: the human needs to express the algorithm in a way that allows the compiler to understand it well enough to be able to really optimize it. And that's where C and C++ fall down. C and C++ are strongly pointer-based languages. The real semantics of almost anything interesting end up involving pretty much unrestricted pointers. In C and C++, there's no such thing as an array - there's just pointers, which you can subscript and a shorthand for pointer arithmetic and indirection(x[n] in C/C++ is the same thing as *(x+n).) That pointer based nature means that in a C or C++ program, it's very hard for a compiler to figure out what things are independent. It comes down to a problem called alias detection. Alias detection is identifying when two variables might be referencing the same location. Alias detection becomes a horrific mess in the presence of unrestricted pointers.

General Comments on Drupal's Error "Unknown table 'n'"

2006-10-25T00:00:00+02:00

Maybe you already know the famous Drupal error “Unknown table ‘n’”. It looks like this:

user warning: Unknown table 'n' in where clause query: SELECT DISTINCT(node.nid), node.type AS node_type, node.title AS node_title, node.changed AS node_changed, users.name AS users_name, users.uid AS users_uid, node.created AS node_created, node_counter.daycount AS node_counter_daycount FROM node node LEFT JOIN users users ON node.uid = users.uid LEFT JOIN node_counter node_counter ON node.nid = node_counter.nid INNER JOIN node_access na ON na.nid = node.nid WHERE (na.grant_view >= 1 AND ((na.gid = -1 AND na.realm = 'all') OR (na.gid = 0 AND na.realm = 'og_public') OR (na.gid = 0 AND na.realm = 'og_all') OR (na.gid = 3 AND na.realm = 'og_subscriber') OR (na.gid = 4 AND na.realm = 'og_subscriber') OR (na.gid = 0 AND na.realm = 'cac_lite'))) AND (n.moderate != 1) AND (node.status = '1') ORDER BY node_counter.daycount DESC LIMIT 0, 5 in /home/orbit42/public_html/includes/database.mysql.inc on line 120.

This error is caused by some module “rewriting” a query of some other module, each of them not aware of each other. This is a powerful Drupal feature, yet it also is highly error prone - as the lots of error alike show. Here’s how to deal with it.

The process of rewriting a query works like this: A module builds a SQL query and passes it to the function db_rewrite_sql, together with the table name and some other information.

In general there are two possible error sources:

A module invokes db_rewrite_sql with wrong parameters
A module wrongly implements the actual rewriting

Each of these errors will occur only if the query is rewritten, which actually may depend on a the module configuration, user state etc. That’s why there are so much different forms of the error.

An example: Module 1 issues a query rewrite, but does it wrong. This will only be discovered when another module really rewrites something. There aren’t to many modules doing this, so it may take a time. The same holds, when a module wrongly implements the query rewriting. In this case, the error will occur within some other module, which is especially nasty.

For this reasons, when dealing with “Unknown table ‘n’”, the list of modules installed is extremely important.

Now let’s dive a little deeper into db_rewrite_sql(). The function is declared like this:

function db_rewrite_sql($query, $primary_table = 'n', $primary_field = 'nid', $args = array())

The first 3 parameters are of interest. First it takes the SQL query as a string, followed by the name of the primary table and the primary id field. Table and id field are supposed to default to ‘n’ and ‘nid’.

‘n’ - does that sound familiar?

Indeed, this ‘n’ causes all the trouble. See, it is assumed that the table queried is called ‘n’ normally, however, this is not required.

This can be an error source when invoking the function. This for example is wrong:

$sql = db_rewrite_sql('Select nid, title from {node} as squirrel');

The primary table name (which correctly should be called the primary alias name) in this case is ‘squirrel’. I like squirrels. Due to the default values, the function is expanded to:

$sql = db_rewrite_sql('Select nid, title from {node} as squirrel', 'n', 'nid');

OK, there is a field ‘nid’, but no ‘n’ anywhere. This is bad.

A correct way to invoke db_rewrite_sql() and keep the squirrel alias is to simply pass it:

$sql = db_rewrite_sql('Select nid, title from {node} as squirrel', 'squirrel', 'nid');

This code should issue no problems - well, unless of course some squirrel hating module handles it wrong.

If a module wants to rewrite queries of other modules, it has to provide a so-called hook. A hook is named [module-name]_[hook_name]. So if your module file is named squirrel_hater.module, the hook would be named squirrel_hater_db_rewrite_sql(). This function takes exactly the same arguments than the original db_rewrite_sql():

squirrel_hater_db_rewrite_sql($query, $primary_table, $primary_field, $args)

Let’s have a look at a real word example. Code like this (a patch taken from here) is a true troublemaker:

/**
 * Implementation of hook_db_rewrite_sql
 */
function content_db_rewrite_sql($query, $primary_table, $primary_field, $args) {
  global $user;
  switch ($primary_field) {
    case 'nid':
      if ($user->uid != 1) {
        $return = array();
        $where = array("n.type <> ''");
        foreach (content_types() as $name => $type) {
          if (!user_access('view '. $name .' content')) {
            $where[] = "n.type <> '$name'";
          }
        }
        $return['where'] = join(' AND ', $where);
        return $return;
     }
     break;
  }
}

Note that this codes checks if the element queried is a node. Therefore it looks if the query deals with a node’s id (case “nid”). This is totally correct.

But then the code assumes the table alias is “n” - for exampel in this line:

$where = array("n.type <> ''");

Notice “n.type”? No squirrels! This is what will be causing an error “unknown table ‘n’” error.

Correct code would use the table alias $primary_table, like so:

 
$where = array($primary_table . ".type <> ''");

Here’s the whole snippet, corrected:

/**
 * Implementation of hook_db_rewrite_sql
 */
function content_db_rewrite_sql($query, $primary_table, $primary_field, $args) {
  global $user;
  switch ($primary_field) {
    case 'nid':
      if ($user->uid != 1) {
        $return = array();
        $where = array($primary_table . ".type <> ''");
        foreach (content_types() as $name => $type) {
          if (!user_access('view '. $name .' content')) {
            $where[] = $primary_table . ".type <> '$name'";
          }
        }
        $return['where'] = join(' AND ', $where);
        return $return;
     }
     break;
  }
}

This little error in the implementation of the db_rewrite_sql-hook is not noticed, as long as the primary table is “n”, which it is in about 95% of all cases. There are, however, some modules, who name the table differently. Not “squirrel”, of course, but for “node”, for example. Additionally, the rewriting is not done for administrators. This turns the code into a neat little time bomb.

There are hundreds of modules, and any of them may contain a wrong implementation like the above one. Additionally there are hundreds of patches, which - like the example - may contain errors, too.

So, my advise in case of trouble is to look for code fragments like above. Look for functions named “[modulename]_db_rewrite_sql” and see if there’s a hard coded “n.[fieldname]” in it.

Note: This article is based on what I wrote on this bug report. I polished it a bit, tried to simplify it here and there, and added some squirrels.

Focus first Form field with jQuery

2006-10-19T00:00:00+02:00

I somehow like the jQuery JavaScript library. Have a look at the jQuery code for the Zebra-Table-Showdown: Ain’t this a beauty?

However, when it comes to focusing the first (visible) field on a form, things are not that elegant. I got it working, though. Here’s my current attempt:

// Focus first element

$.fn.focus_first = function() {
  var elem = $('input:visible', this).get(0);
  var select = $('select:visible', this).get(0);
  if (select && elem) {
    if (select.offsetTop < elem.offsetTop) {
      elem = select;
    }
  }
  var textarea = $('textarea:visible', this).get(0);
  if (textarea && elem) {
    if (textarea.offsetTop < elem.offsetTop) {
      elem = textarea;
    }
  }
  
  if (elem) {
    elem.focus();
  }
  return this;
}

Use it like this:

$(window).load(function() {
  $('form').focus_first();
);

I wonder, though, if there is a more comprehensive expression to find the first element of either type “input”, “select”, or “textarea”.

$('input:visible,select:visible,textarea:visible', this).eq(0)

will not work, since it will always return the first “input”, regardless if there is a “select” or “textarea” before it.

Any suggestions?

std::clock(), threading, and Linux

2006-09-09T00:00:00+02:00

I had a strange experience with the boost::timer class the other day, since it never elapsed within a certain thread. As it turns out, this was because std::clock() itself doesn’t tick but returns the same value over and over again - a value that probably indicates the time the tread was started. The solution is rather simple: I implemented the boost::timer interface based upon boost::xtime from the boost threading library like this:

class XTimer {
public:
  XTimer() {
    xtime_get(&_start_time, boost::TIME_UTC);
  }

  XTimer(const XTimer& other)
  : _start_time(other._start_time)
  {
  }

  ~XTimer() {
  }

  double elapsed() const {
    boost::xtime now;
    xtime_get(&now, boost::TIME_UTC);
    return boost::lexical_cast<double>(now.sec - _start_time.sec);
  }

private:
  boost::xtime _start_time;
};

XMLRPC value types and C++

2006-08-05T00:00:00+02:00

A while ago, I was looking for a C/C++ XMLRPC client implementation. However, for some reason or another, I was unsatisfied with the recent implementations, which mostly seem utterly complex for my purposes. That’s why I decided to roll out my own. While the XML building and parsing is more or less straight forward (although there’s too much redundancy and unneccessary tags in the spec for my taste), implementing the possible value types made me scratch my head. The XMLRPC protocol supports a set of data types: integer, string, double etc., but also an array or a struct. Integer and double are native types, string is either a char pointer or a class. How this should be put within one class? My first sketch introduced a quite complex model with an impressing inheritence chain, but then boost::variant came to the rescue:

/*! \brief XmlRpcValue is defined as a variant that can
 *         take string, int, double, bool,
 *         and collections of itself, namely a vector and a map
 */
typedef boost::make_recursive_variant<
    std::string,
    int,
    bool,
    double,
    std::map< std::string, boost::recursive_variant_ >,
    std::vector< boost::recursive_variant_ >
>::type XmlRpcValue;

/*! \brief Key and value pairs form a struct
 */
typedef std::map< std::string, XmlRpcValue > XmlRpcStruct;

/*! \brief A list forms an array 
 */
typedef std::vector< XmlRpcValue > XmlRpcArray;

This defines a type called XmlRpcValue. This type can contain either a string, an int, a bool, a double, or a vector of XmlRpcValues or a string-based map of XmlRpcValues. Isn’t that cool?

We can easily utilize this type like this:

/*! \brief convert given parser value into xmlrpc value.
 *
 * \exception boost::lecial_cast_error
 *              Conversion wasn't successful
 * \exception std::invalid-argument
 *              Data type not recognized
 */
static XmlRpcValue _read_parser_value(const ParserValue& val) throw (std::exception) {
    // Check all types

    if (val->type == "int" || val->type == "i4") {
        return XmlRpcValue(boost::lexical_cast<int>(val->value));
    }
    if (val->type == "double") {
        return XmlRpcValue(boost::lexical_cast<double>(val->value));
    }
    [...]
    // Unrecognized data type

    throw std::invalid_argument("Unknown data type detected");
}

ParserValue in this case is a simple structure containing information about parsed data:

/*! \brief A single value extracted from response
 */
struct ParsedValue {
public:
    /*! \brief The name. Used only for struct members */
    std::string name;
    /*! \brief The value as string */
    std::string value;
    /*! \brief The type, value type tag name like
     *         "int", "struct", "array" etc. */
    std::string type;
};

A triangle with rounded corners in C#

2006-01-29T00:00:00+01:00

For one of my projects I needed to draw triangles with rounded egdes, and since I didn’t found anything about this topic elsewhere, I had to figure it out by myself. Luckily no mathematics is needed, except the Pythagorean theorem. So here’S how I did it, complete with a sample implementation using C#.

To make things easier, I started with a right-angled triangle where the two sides forming the right angle are vertical and horizontal. This makes applying a rounded corner for the right angle easy, since it is a common problem known from drawing rounded rectangles. You’ll find examples for this all over the net. I instead will discuss, how to round the two other edges, which both form an angle of 45 degrees.

Here’s a sketch demonstrating the problem. A circle with radius r needs to be placed at point C for the two sides forming the triangle becoming its tangents. How do we calculate C? Well, we already know that the length of BC is r, so we just need to figure out the length of AB. AB itself has the same length then BD, whitch is the sum of BC and CD, where BC is r:

AB = BD
AB = BC + CD
AB = r + CD

Now the points C, D, and E form another right-angled triangle with CE = ED = r. So we can apply the Pythagorean theorem:

AB = r + CD
AB = r + sqr(CE * CE + ED * ED)
AB = r + sqr(r * r + r * r)
AB = r + sqr(2 * r * r)
AB = r + sqr(2) * r
AB = (1 + sqr(2)) * r

For drawing circles or parts of it in C#, we need to provide the surrounding rectangle R, which obviously starts at Rx = Cx - r and Ry = Cy - r.

Rx = Cx - r = Ax
Ry = Cy - r = Ay + (1 + sqr(2)) * r - r
Ry = Ay + (1 - 1 + sqr(2)) * r
Ry = Ay + sqr(2) * r

Here’s the C# code to contruct a rounded rectangle’s graphic path. The right angle is rounded with double radius, just because it looks better:

// Build it in default size, pointing to left bottom,

// so 90° angle is left bottom.

int radius = 10;
int widthArcRect = 2 * radius;
int left = 0;
int right = 100;
int bottom = right;
int top = 0;
int cornerDist = (int)(Math.Sqrt(2) * radius);
GraphicsPath p = new GraphicsPath();
// bottom left corner (90°)

p.AddArc(left, bottom - 2 * widthArcRect, 2 * widthArcRect, 2 * widthArcRect, 90, 90);
// left line

p.AddLine(left, bottom - 2 * widthArcRect, left, top + cornerDist + widthArcRect);
// top corner

p.AddArc(left, top + cornerDist, widthArcRect, widthArcRect, 180, 135);
// diagonal line

p.AddLine(left + cornerDist, top + cornerDist, right - cornerDist, bottom - cornerDist);
// right corner

p.AddArc(right - cornerDist - widthArcRect, bottom - widthArcRect, widthArcRect, widthArcRect, 315, 135);
// Bottom line

p.AddLine(right - cornerDist - widthArcRect, bottom, left + 2 * widthArcRect, bottom);
p.CloseFigure();

Now having the triangle, we can transform it, to point in any directions and to have any size. If for example we want to draw a triangle pointing to the right beeing 20 pixel in height, we apply the following transformation:

int newHeight = 20;
Matrix m = new Matrix();
// Rotate

PointF center = new PointF((right - left)/ 2, (bottom - top) /2 );
m.RotateAt(-135, center);
// Rotated triangle is larger than the original

int widthOriginal = (right - left);
float lengthHypotenuse = float(Math.Sqrt(2) * width);
m.Translate(- width / 2, (lengthHypotenuse - width) / 2);
// Scale to fit size

float scale = (float)(newSize / lengthHypotenuse);
m.Scale(scale, scale);
// Apply Matrix

p.Transform(m);

What is happening here? After rotating, the triangle’s bounding rectangle is higher then the original, since the hypotenuse is now vertical. The length of the hypotenuse (Lh) is:

Lh = sqr(width * width + width * width)
Lh = sqr(2 * width * width)
Lh = sqr(2) * width;</code>

We now have to move the triangle down a bit, so that its top corner is within the orignals path boundaries (This makes positioning much easier). The amount we have to move is the lengh of the hypotenuse minus the width divided by two. We also have to move the whole triangle to the left, since it was rotated at its bounding rectangle’s center, which now forms the left side of the triangle. Afterwards the triangle is scaled, since we want the hypotenuse to be 20 pixel in height.

After the transformation, we have a right angled triangle with rounded edges, pointing to the right, and 20 pixel in height. We now can apply a last transformation to place it at the coordinates (5, 10), for example:

m.Translate(5, 10);

Microformats: Prefering machines over people

2006-01-17T00:00:00+01:00

Anybody ever took a look at the Datetime Design Pattern at Microformats.org (via Georg Bauer (German))? It goes like this:

<abbr class="foo" title="YYYY-MM-DDTHH:MM:SS+ZZ:ZZ">Date Time</abbr>

I think this is a misuse of the abbreviation tag - and worse: it’s completely ignoring user experience. Most nowadays browsers display the text of the title attribute when you hover over the abbreviation. Now imaging a user browsing your microformatted page. When he hovers over a date like - say - May 4th 1970, which is my birthday, he’ll see a tooltip stating “1970-05-04T18:10:52+01:00”. Doh!

Not to tell that screen readers usually read the abbreviation’s title. So, a screen reader would read “One Nine Seven Zero Minus Zero Five Minus Zero Four T One Eight Colon…” Really well done, you microformat people.

Like Colin Lieberman writes on A List Apart:

The point of including expanded forms of abbreviations is to make sure that users can access those forms if needed. The value of the abbr element lies in its ability to tell screen readers what they should read (thus improving the flow of spoken text), while also providing additional useful information to visual users with learning or cognitive disabilities (or those coping with language barriers). The assertion that abbr is structural is misguided, as the point of the tag is the content of its title attribute.

He’s right. Abbreviation is not structural. Even if it once was intended to be, in real life it isn’t. It seems, this has been forgotten by the microformat astronauts.

Don't trust PHP_SELF

2006-01-11T00:00:00+01:00

Sean Coates points out why $_SERVER[“PHP_SELF”] must be considered unsafe. You should never use it without at least converting it to entities:

print htmlentities($_SERVER['PHP_SELF']);

This offers me the chance to again repeat one of my mantras: Create a function for everything, at least when reading values from somewhere, even for something simple like retrieving the value of PHP_SELF. The reason is maintenance: If you haven’t previously validated the value of PHP_SELF there will be only one place to change, instead of checking all your code.

Here’s my implementation:

/**
 * Static returns the recent URL.
 *
 * @param Boolean Indicated if query string (?...) should be included or not
 * @returns String
 */
function recentURL($includeParams) {
  $http = "";
  if ($includeParams)
  {
    $http = $_SERVER["REQUEST_URI"];
  } else {
    $http = $_SERVER["PHP_SELF"];      
  }
  return toSafeURL($http);
}
/**
 * Static. Processes an URL to strip of possible injection code 
 * 
 * @param String The URL to process 
 * @return String The cleaned URL
 */
function toSafeURL($url) {
  return htmlentities(strip_tags($url));
}

Large-Scale C++ Software Design

2005-12-03T00:00:00+01:00

Following up on a comment from Udi Dahan about physical design Len Holgate points out that

If you're working in C++ and you haven't read Large-Scale C++ Software Design (APC) by John Lakos then you really should.

This is definitly true. John Lakos’ book - beeing as far as I know the only one on the topic - is a must-read for each C++ developer.

Len is using Lakos’ book as an argument against my post about C++ still beeing needed? and further states:

You have to actively manage your dependencies all the time, no matter what language you're working in. If you don't do it in C++ you can end up in a hell of slow compiles and horribly entwined code that all needs to be rebuilt if you ever change one class. If you do it right you rebuild just what needs to be built and you depend on just what you need, and no more.

Again: true. However, each project contains some very fundamental libraries, nearly all other packages depend on. A custom string class comes to mind ;-), but there are other. Let me tell you a little bit about the application I’m developing at work to get things clear.

We follow a model driven development approach. This is: One should be able to model what we call domains in any UML-capable CASE tool from which some basic code is generated so that the new domain can be used right away without further manual work. This allows easy prototyping of new business domains and it futher allows adjustments of the application’s class model at runtime.

To achieve this, we needed to bring reflection to C++. Reflection - known from most modern languages like C#, Java, Ruby etc. - allows retrieving class level information from instances. .NET’s GetType() function is an example of this. Our reflection model itself is an implementation of the UML 2.0 superstructure.

We established three meta modelling levels M0 (instance level), M1 (reflection) and M2 (runtime modelling). For example, to participate from reflection, all business objects must be derived from a class called M0Instance.

At this point, it should be quite clear, that nearly all of our code in some way or another depends on what we called the Kernel, containing the levels M0 and M1. We of course applied the guidelines of Large Scale C++ Software Design, but still changes to any of the Kernel’s core interfaces lead to a massive recompile. Just to give you some numbers: 30 Minutes on Windows, 1 hour on Linux and 2 hours on HP-UX.

I was arguing that development in C++ takes a significant higher amount of time compared to other high level languages, and that the long compile time is one factor among others. I think this still holds. Of course, changes in basic packages should occur rather seldom, however, our application is evolving, and so is the Kernel.

As a side note, the massive compile time established a culture of “better don’t touch”. I was realizing this, when I found myself adding complicated, ugly, and similiar code (referenced further as “the shame”) in several application layers when extending the application’s user access control features. By just introducing one new function to the reflection level, all this code was made obsolete. I was thinking about this function when I added the shame the first time, I also thought about it the second time. I finally did it in the middle of adding the shame the third time…

Some last note: I was implementing the Kernel in C#, mainly to learn the language. This, including persistence, two simple domains and a basic generic UI framework, took me four to five days. The initial C++ Kernel development - without any UI stuff - took several man months.

Is C++ still needed?

2005-11-24T00:00:00+01:00

Christopher Baus is reasoning about why C++ is falling out of favor:

In domains where low level access is critical, developers are going for C. If developers can live with higher level abstractions, then they are skipping straight to Java or Python or Ruby or PHP, and by passing C++ altogether. C++ is getting squeezed out in the middle.

I think he’s right. If you’re doing large scale business applications, like I do on work, C++ definitely is not the language of choice. Some reasons:

C++ is slow. Amazing, but true. We evaluated both Java, C# and C++, and Java and C# were up to five times faster on usual use cases, like retrieving and instantiating objects from a database. This is because creating a new instance in memory takes much more time in C++ compared to other, even interpreted, languages - unless you don't write your own memory management. But as Christoper explains, C++ is missing some low level feature in this area, too ;-).
C++ takes more development time. Not only because you have to write your own memory management, there's much more to take care about in C++ than in other high level languages, mostly due to missing garbage collection. And a lot of time is consumed by compiling, which takes significantly longer than in any other language (talking about hours compared to minutes, actually).
C++ isn't open to other languages. While you can bridge from say Python to Java or from Java to C, C++ binaries cannot be accessed (unless they publish core C-functions). This is because there is no standard on how C++ should publish its library entry points (don't know if this is the right term), so each compiler does it slighly different.
C++ doesn't support module oriented development very well. For two reasons: First the interface definitions (the .h files) are cluttered with implementation stuff like private variables. If you work around this by holding your data in a separate structure, you pay with an additional memory allocation, which slows the application down. Second, C++ misses a concept like C# offers with "internal" (Java and VB offer same concepts): Code that is public within a module, but private from outside.

Nevertheless, we chose C++ as our language. Why? Because it was the only language that (using the QT framework) ran on Windows, Linux, and several (already outdated) *nix platforms, like for example HP-UX 20. Platforms where Java is available in version 1.0 only, if available at all. Platforms where Mono will never run. This however rather is a niche and more and more of these platforms are replaced by Linux nowadays.

Anyhow, there are cases were C++ comes in handy: All the high level languages and their libraries offer you about 90 percent of what you will need in everyday business programming. But if you want some advanced stuff like the most current juicy UI features or access to the systems network layer, you are left alone. There are reasons for this (for example the .NET framework must be compatible to Windows 98), of course and most languages will allow to use the system’s API, which is C.

However, using for example the Windows API from C# is a pain in the ass, since you must redeclare each and every structure and function. You must deal with pointers or pointers to pointers etc. All of which is native to C++ and saves you a bunch of time (and results in much cleaner code), as this comparison shows. In a recent project of mine I therefore happily mixed C++ and C#, the same as I did back when I was programming in Visual Basic 6.

You may note however, that I’m doing this because I can. C++.NET managed code can be used by C# without any problems (Dude, I really love it). For Visual Basic I created COM objects in C++ using ATL. But what if Java was my main language or Python? I’d be stuck (unless I’m doing CORBRA, which - well - isn’t really made for this kind of thing).

Conclusion? C++ sometimes is the best choice, sometimes not. If C++ is falling out of favor, then also because it didn’t care about the rest of the world - and still doesn’t care.

Sending spam through contact forms

2005-09-15T00:00:00+02:00

There’s a wave of contact form spamming going around and it hit the Comic Marktplatz, too. The main purpose of this attack is to figure out websites providing forms that can be misused to send spam, effectivly turning them into open relays. This affects not only contact forms, but also blog comments.

Anders Brownworth has the details about all this (also see his blog entry). However his solution for PHP has some syntax errors. These are corrected by David Seah.

Shortly said, you should remove line feeds from every user entered string that becomes part of a mail’s header. These usually are

The sender (From:)
The subject (Subject:)

David also suggests to remove common header tokens like “to:”, “subject:”, “mime-type:” etc. from this strings. I personally think, this is a good idea. I turned David’s code into a neat function, replacing the original preg_replace with str_replace, for performance purposes:

<?php
/**
 * Clears header field to avoid injection
 * http://www.anders.com/projects/sysadmin/formPostHijacking/
 * http://www.davidseah.com/archives/2005/09/01/wp-contact-form-spam-attack/
 */
function preprocessHeaderField($value)
{
  //Remove line feeds

  $ret = str_replace("\r", "", $value);
  $ret = str_replace("\n", "", $ret);

  // Remove injected headers

  $find = array("/bcc\:/i", 
                "/Content\-Type\:/i", 
                "/Mime\-Type\:/i", 
                "/cc\:/i", 
                "/to\:/i");
  $ret = preg_replace($find, 
                      "**bogus header removed**",
                      $ret);

  return $ret;
}

David also suggest to do this with the message text itself, which I personally regard as not beeing necessary, since the message text usually can do no harm.

It seems the robots are systematically scanning websites for input fields to provide an email, name and message. That’s why they also try this out on comment forms. Since these forms usually also send a mail to the post author, they are also vunerable. So you better watch for updates of your blog software.

Which finally brings me to one of my favourite topics: Architecture. Regarding the Comic Marktplatz, I needed to apply the fix to exactly one place, since all mailing is done through one single class representing a mail. No need to look up every possible vulnerable form throughout the application.

Compare this - just for example and without any judging - to my blog software, Wordpress 1.2 (yes, still… upgrading takes too much time). Do a search for “@mail” in the wordpress source files and you will find several occurances, all of whom potentially are vulnerable. There are even more including the plugins.

In Wordpress 1.5, this hasn’t been quite corrected. Though there is central function called wp-mail(), it acts just as a very small wrapper around the PHP mail() function. Basically it only adds a content type and MIME version. It howevers still requires the caller to prepare a basic header including the sender (From:), which exactly is where the attack takes place. All this calling code must be identified and fixed to secure the application, which is error prone.

Morale? Code duplication is bad, centralization is good. Think before you hack. Teach this, preach this!

PS: I always wondered why PHP mail() doesn’t take the sender as a parameter - this seems weird.

Update: There’s an article on Secure PHP investigating the full story.

Safari first to pass ACID2 test

2005-04-29T00:00:00+02:00

Safari is the first browser to pass the ACID2 browser compliance test. Respect! rm

Tutorial XHTML2 and XForms

2005-04-20T00:00:00+02:00

Last event in my “Do as much in a couple of days as you would normally do in a year”-week: Together with Natalia and Catarina I attended Steven Pemberton’s Tutorial on XHTML2 and XForms, organized by the German and Austrian section of W3C.

Pemberton is a brilliant speaker, easy to follow for both unexperienced and experienced users, and entertaining, too. He covered the basics of XHTML2 and explained why things were decided the way they were. The recommendation looks quite promising. However, of course, it will be of pratical relevance in about five to six years from now - if at least.

However, XForms is already in use, as Pemberton explained. He demonstrated some XForms-applications, and it was really amazing what already can be done. This is something, I’ll have a deeper look at the other day. The people at Mozilla are working on it, so maybe it will becoming wider spread soon.

Finally, I’d like to thank the German and Austrian section of W3C for this event. You did a great job!

Beware of Apache's Multiviews

2005-04-15T00:00:00+02:00

Releasing a web project for the first time brings up something new every day. The lesson I learned yesterday was to never rely on Apache’s Multiview feature, since it doesn’t work with the most important browser on earth: Googlebot.

As you may know, Google does deep indexing of new content onl once a month, which happened to be yesterday. But imagine my horror, when I discovered each of my content pages delivering an 406 error to Googlebot. Oh me, oh my! What has happened here?

It turned out that this is a well-known problem of Apache’s multiview feature. Multiviews allow substitutions of file extensions, so you can call an URL like www.somehost.org/mypage.php using www.somehost.org/mypage. Apache will figure out there is only on file (mypage.php) matching the request and will serve this instead returning an error 404 - not found.

So far, so good. And handy, indeed. But…

While doing this, Apache will check if the MIME-type of the file used for substitution matches the MIME types accepted by the client that does the request. This ain’t a problem for most browsers, since they usually accept any MIME type. However, Google decided to accept only (or mainly) text/html.

Well, mypage.php creates text/html, doesn’t it? So where’s the problem?

The problem is, that while it may create text/html, mypage.php itself is of MIME type application/x-httpd-php. This type is not accepted by Googlebot, hence the error 406 - Not accepted.

Dooh!

Unfortunately, there doesn’t seem to be a general solution for Apache 1.3.x (while there is one for Apache 2). And since I wasn’t feeling lucky enough to figure out by myself, I decided to rely on Apache’s rewriting capabilities. Which, to be honest, I recently tried to avoid, since regular expression usually give me headaches.

My first hotfix was to simple hardcode every page:

Options -Multiviews
RewriteEngine On
RewriteBase / 
RewriteRule ^mypage/(.*) /mypage.php/$1
RewriteRule ^mypage$ /mypage.php

This simply will rewrite /mypage/some/parameters to /mypage.php/some/parameters (the first rule) or /mypage to /mypage.php (the second one). Note that parameters like ?arg1=some&arg2=parameter are handled by Apache automatically. So something like /mypage?arg1=some&arg2=parameter will be handled by the second rule.

This hotfix made Googlebot happy, and therefore me, too. However, I came up with a more general solution that covers all cases:

Options -Multiviews
RewriteEngine On
RewriteBase /

# If there is no . before / rewrite to $1.php/$2

RewriteCond $1 !images|downloads
RewriteRule ^([^&#92;./]+)/(.*) /$1.php/$2 [L]

# Find single files like mypage

RewriteCond $1 !images|downloads
RewriteRule ^([^&#92;.]+)$ /$1.php [L]

First, all the rules should not apply to real directories, like for example images or downloads. Thats why I excluded them using a rewrite condition. This is far from good, and I hope to find a general (and performant) solution for this, too. But since regular expressions give me headaches, I won’t do too much at a time of them.

The first rule now checks for a request that doesn’t contain a dot before the first slash, which means there is no file extension provided. It must have at least one char before the slash, though, else there will be a problem with the root directory “/”. The rule will simply append “.php” and exit.

The second rule does nearly the same for requests like “/mypage”, where there is no trailing slash.

I’m quite sure both cases can be put into one single rule, but as you may know, regular expressions give me headaches. So this is left up for another time.

Replacing Inheritance With Composition

2005-01-31T00:00:00+01:00

Ivan Moore briefly discusses replacing inheritance with composition regarding the amout of testing code required. He concludes that with composition,

instead of having the tests duplicated (or using inheritance in the tests to remove the duplication) the tests can now be split into two parts. One test for the class with the code that is common, and separate tests for the code that is different.

This is quite similiar to what I wrote about applying filters to a collection. And as Ivan notes

There are other benefits in addition to the simplification of the tests, and the removal of duplication. Using composition also allows more flexibility in combining these classes together.

Less PHP Database Code Using The Factory Design Pattern

2005-01-27T00:00:00+01:00

It’s PHP design pattern time again! Today I’ll show a simple way to

Encapsulate database access in a class
Make queries return objects rather than records or arrays
Reduce code to process database results

All this is achieved by using the factory pattern. Read on!

It usually is a good idea to encapsulate database access into a class. Not necessarily because you may change your database more easily (since for most PHP applications the database will be MySQL for still a long time, due to provider configurations), but to get rid of all this repeating database access code, which tends to be error prone.

If you start to wrap db code fragments into a class, however, you’ll notice that having hidden MySQL inside a neat little class, you still have code like

while ($record = mysql_fetch_array($dbHandle)) doSomething($record);

spread all over your application. And that’s the last thing you want, actually. So it’s time for another abstraction, a recordset, is it?

Well, not necessarily. Try using the factory design pattern.

The idea of the factory pattern is to use a function to create an instance of a class rather than calling the constructor using new. This is because a function can be overridden, while a constructor cannot. This makes quite a difference, since for using the new keyword, we must know the concrete type of the instance to create. Having a function, this is not required, since it is enough to know the function exists.

Let’s start with a simple class like this one:

class Person {
  var $firstName = "";
  var $lastName = "";
  var $age = 0;
  
  function Person($firstName, $lastName, $age) {
    $this->firstName = $firstName;
    $this->lastName = $lastName;
    $this->age = $age;
  }
}

Now we write a factory. It is really simple, since it contains just one function. We pass a record, which actually is an associative array. This is a brief abstraction in this case. Here we go:

class PersonFactory {
  function &create(&$record) {
    return new Person(
      $record['firstname'],
      $record['lastname'],
      $record['age']
    );
  }
}

Simple, isn’t it? This of course assumes the fields in the database are called accordingly.

Now on to the database wrapper class. Having a factory class, a database query can return objects directly, rather then just records or a query handle:

class DB {
  // .. some other code to open connection etc.

  
  // returns an array of instances

  function &queryArray($sql, &$factory) {
    $queryID = mysql_query($sql);
    $ret = array();
    if ($queryID) {
      while ( $record = mysql_fetch_array($queryID) ) {
        $ret[] = $factory->create($record);
      }
      mysql_free_result($queryID);
    }
    return $ret;
  }
  
  // Returns a single instance	

  function &querySingle($sql, &$factory) {
    $queryID = mysql_query($sql);		
    if ($queryID) {
      if ( $record = mysql_fetch_array($queryID) ) {
        mysql_free_result($queryID);
        return $factory->create($record);
      }
    }
    return false;	
  }
}

I hope you get the idea. There is no magic in this code, everything is straight forward. Having the DB returning an array of instances makes it much easier to iterate through it, using a common foreach loop.

What’s missing is a function to load objects from database. Here are two simple implementations:

class PersonDB {
  // returns one instance

  function &findByID($id) {
    return DB::querySingle(
      'Select * from Person where id = $id',
      new PersonFactory()
    );
  }
  
  // returns an array

  function &findByAge($age) {
    return DB::queryArray(
      'Select * from Person where age = $age',
      new PersonFactory()
    );
  }
}

And finally here’s the client code that uses it all:

//print all persons that are 35 years old

if ( $arr = PersonDB::findByAge(35) ) {
  foreach($arr as $person) {
    echo $person->foreName . ' ' . $person->lastName . '&#92;n';
  }
}

Neat, isn’t it? Notice that each of the functions and classes presented are simple and more or less self-explaining. Combined, however, they provide a great flexibility. Now compare this to the avarage PHP database code, you’ll find in oh so many PHP applications!

Applying Arbitrary Custom Filters To A Collection: Chain Of Responsibility

2005-01-21T00:00:00+01:00

I faced some annoyance of my Randomized Blogroll Plugin the last days: My own blogroll contains some subscriptions to ego searches that I don’t want to publicize on my webpage, since they are

depressing, and
not of any public interest.

Getting tired of removing them by hand everytime I update my blogroll, I started to implement filtering. Here’s what I came up with. See my mind at work.

My first sketch was to create a simple class called Filter, containing just one function that gets passed a collection, iterates through it and does what needs to be done. This is PHP 4.x code:

class Filter {
  function filter(&$collection) {
    // Iterate and apply filter 

  }
}

However, if I want to apply an additional filter (renaming root category “Subscriptions” to “General”, for example), the collection is stepped through a second time: ugly.

So the next step was to extract the iteration from Filter and move it into the collection class:

class Collection
{
  function filter(&$filter) {
    foreach ([myvalues] as $item)
      $filter->filter($item, $this);
  }
}
  
class Filter {
  function filter(&$item, &$collection) {
    // apply filter on $item

  }
}

However, this doesn’t solve the problem of applying more than one filter, since collection->filter() still only takes one Filter instance. Maybe I should make it accepting an array, too?

No. There is a nice little design pattern at hand that solves this problem: chain of responsibility. This pattern can be applied if

the set of objects that can handle a request should be specified dynamically.

This is excatly what I want to do. Applying this pattern the collection can be left untouched, while I extend the Filter class:

class FilterChain {
  /**
   * The next chain mamber
   */
  var $nextInChain = false;
  	
  /**
   * Append new member to the end of the chain
   */
  function appendToChain($member) {
    if ($this->nextInChain == false) {
      $this->nextInChain = $member;
    } else {
      $this->nextInChain->appendToChain($member);
    }
  }
  	
  /** 
   * Filter
   */
  function filter(&$item, &$collection) {
    if (
      $this->doFilter($item, $collection) &&
      $this->nextInChain
    ) {
      $this->nextInChain->filter($item, $collection);
    }
  }
  	
  /**
   * Does filtering. Overridden in subclasses.
   * Returns TRUE if next filter can be applied, FALSE if not (the item has been removed)
   */
  function doFilter(&$item, &$collection) {
    return false; // default. Override this.

  }
}
  
class Filter extends FilterChain {
  function doFilter(&$item, &$collection) {
    // apply filter on $item 

    // return FALSE if item was removed, otherwise TRUE

  }
}

The code is simple. FilterChain delegates filtering to doFilter() which would be virtual protected in C++ or comparable languages. appendToChain() find the last member of the chain and sets its next element property.

The usage is simple, too:

$filter = new SomeFilter();
$filter->appendToChain( new SomeOtherFilter() );
$someCollection->filter($filter);

Reflecting the state of affairs for this plugin, I have customizable input, customizable output, customizable filtering and I’m starting on customizable sorting. Maybe it neds some renaming? What about The most customizable blogroll plugin ever?

Building A Better C++ String Class?

2005-01-18T00:00:00+01:00

Preston L. Bannister tries to build a better C++ string class than provided by the C++ standard libraries (found through Christopher Baus). However, I think he will fail, since he’s too much involved in low level things, rather then design.

Preston states that

I seldom use the C++ string classes supplied with the standard libraries, out of a belief that (for my purposes) that I can easily do better, building on bits of history and experience. Considerations that go into building a simple and highly efficient C++ string class include the following.
The standard <string .h> supplied C string functions are highly optimized (as described in a previous post).

Heap operations are best avoided (where practical) in long running applications. Use of a free list is very effective when the application has a cyclic workload (as most do).

Use of C++ inline class methods allows both readable code and most effective use of the optimizer.

I actually wonder why he’s so negative about the standard C++ string class, and what makes him belief the standard implementation isn’t highly efficient. Let’s take a look at the Visual C++ implementation, for example. Doesn’t it fullfill, what Preston considers?

It uses calls to the supplied C string functions, which turn out to perform very well on Windows machines, as he himself figured out.
It avoids unnecessary heap allocation by implementing copy-on-write, allocating in blocks and similiar tricks.
Since it is a template class, all functions are inlined.

Remarkably, the string class gains perfomance by changing the point of view from low level operations like memory copying to the actual usage. Take for example the copy-on-write strategy (Scott Meyers wrote about it in “More Effective C++”, Rule 29). This is a true performance boost, since you avoid memory beeing copied on each assignment or copy construction operation. And have in mind that each function returning a string class implicitly invokes the copy constructor. In other words: This is a very common use case for a string class.

Figuring out the best way to copy an array of chars is simple. But designing a string class is much more than this: It is about avoiding copying whereever possible, too. And I doubt if the string class Preston is to come up with, will be look too much different from the current standard implementation in the end.

PHP Performance Also Is A Question Of Design

2005-01-14T00:00:00+01:00

Harry Fuecks, who runs the excelent phpPatterns site talks about object oriented programming and performance . He states that procedural PHP leads to slower apps and challenges everyone to prove the opposite.

Harry argues, the argument of procedural code beeing faster

is based on benchmarking code. What it ignores though is the human aspect of writing code and that's where a side-effect of OOP can result in improved performance.

Bearing in mind I'm talking only in a general sense, I think the tendency with procedural code is to use "brute force" - the actual meaning and behaviour of the logic being masked by the spaghetti. If what the code was actually doing was transparent to the developer, they might see some of the "blinding" inefficiencies they've introduced.

I totally agree. As I already argued regarding C++ optimization, performance is rather a question of design, not code.

In Addition to C++ Optimization

2005-01-03T00:00:00+01:00

In addition to the recent discussion about C++ optimization: In this CodeGuru Forums thread people talk about optimizing conditional branches like

if (a == 0) x = c;

Interesting, sure, though I still agree with this post, which argues that:

If there was a REAL performance benefit to be made, don't you think the compiler designers would have added this in their optimizer already? Compiler benchmarks are a big issue, and they add a LOT more complicated type optimisations than this. Most of the time, if you want to get better performance, you have to rethink the program at a higher level. Put your brains at work to find a better design, and let the compiler twiddle the bits for you.

Optimization, Templates And The Rest

2004-12-18T00:00:00+01:00

Does low-level optimization really works? Do templates bring speed? Two articles by Raymond Chen and Christopher Baus started a discussion, that I’m about to join. I’m arguing that in most cases, you do not want to do low-level optimization, since the speed you gain isn’t worth the effort, and you should rather concentrate on the design. At least if you are an application developer.

Raymond Chen illustrates that optimization is often counter-intuitive. Well, with the right example, you can prove anything, and Raymond’s example seems rather bizarre to me. Maybe since I’m an application developer that couldn’t care less about low-level details.

Now Christopher Baus raises the question up to a level I’m familiar with. He starts with shortly discussing C++ function inlining and optimization and after that quickly jumps to talk about template meta programming and its impact on performance. Christopher’s says that

It is extremely difficult to optimize for modern processors and results can vary even with in a product line; therefore most application developers should not spend their time trying to optimize the speed of a single operation such as a function call.

I agree with him, that it’s useless to do low level optimizations and spend your time thinking about inlining or not. However, I’d like to give another argument. Even if you can be sure that some optimization really speeds up things, it is still a waste of time in nearly all cases. Why? Because the amount of time your function is faster is way to small to be notable. Think of some inlined accessor and let’s say you safe 10 CPU instructions (I’m too lazy to look up the actual value). On a 1.5 MHz processor, 1.5 Millions instructions are executed a second. Let’s assume that one instruction takes one CPU cycle (indeed, AMD processors are able to execute even several instructions in one CPU cycle), these 10 instructions safe you 0.0666 milliseconds. You therefore need to call this accessor 15,000 times to make your application just one second faster. And you better do this in a loop.

From my experience as an application developer, bottlenecks result far more often from wrong design decisions. For example when using a linked list where a hash table would be more appropriate. If you use a database, defining your indexes correctly is something to worry. However, optimization comes last, and you better let a profiler tell you where to look.

Back to Christopher’s article: He continues arguing that

Recently a new version of the inline argument has re-surfaced. Many proponents of "generic programming" in C++ claim that by binding functions at compile time the resulting executable is faster (less code to execute to make a function call, sound familiar?). I am not convinced. What you do get is BIGGER object code. Bigger binaries are more difficult to cache, and performance on modern processors is all about effectively using the cache.

This may be or may not be, I actually know too little about processors to judge this statement. Nevertheless, I’d like to introduce another argument here: Design. Again, I speak as an application developer, and things may look different if you are programming devices or tricky algorithms.

When you are doing applications, you should concentrate on the design rather then speed. There is a time to optimize and it is at the end, when you know where the bottlenecks are. Your design should be clear, flexible and understandable. It contains the logic of your application, and you will spend a lot of time changing and extending your code, because requirements will change and grow over time. You also will spend a lot of time compiling (I compile after each change, since compiling is the first test the code must pass), so you should consider compile time dependencies. Whether you use template meta programming (which is far more then just using templates here and there) or not, should be primary a design decision.

Christopher ends up with some ranther ranty comment about template meta programming in generel (and is second by Len Holgate):

Template programming has become a mental game for those who want to be C++ giants. It is a proving ground. C++ templates are not a meta-programming language. Those that use templates for advanced meta programming are abusing a side effect of a language construct. The fact that the syntax is so obtuse (because it was never intend to be used in this way) provides a way to prove mental prowess.

Well, maybe. I tend to believe that we never will see large and complex business applications build entirely on templates. However, template meta programming can be useful sometimes, and you can do neat little things. Think of Andrei Alexandrescu’s Modern C++ Design, for example.