Document version 4

1 files changed, 12 insertions, 2 deletions

diff --git a/README b/README
index a05d0de..ea504f6 100644
--- a/README
+++ b/README
@@ -67,10 +67,20 @@ Now we look at the continuation entry parts: A column_id of 0x4003 denotes a sta
 version 3
 =========
 
-Some of the files on disc are obfuscated while others are plain. They consist of chunks of lha compressed data (as readable by the lha command). The obfuscation is a simple XOR of the first 32 (for streets: 34) bytes with a static 4 byte key that changes with every CD. Since lha headers do have a static signature (i.e. -lh5-), it's easy to derive the current key by xoring the static value with the bytes found in the file. Conveniently un-obfuscated files generate a all-zero key and don't need special treatment. The path name of each chunk is identical so it must be re-written and the header checksum must be re-calculated.
+All strings are iso8859-1. Some of the files on disc are obfuscated while others are plain. They consist of chunks of lha compressed data (as readable by the lha command). The obfuscation is a simple XOR of the first 32 (for streets: 34) bytes with a static 4 byte key that changes with every CD. Since lha headers do have a static signature (i.e. -lh5-), it's easy to derive the current key by xoring the static value with the bytes found in the file. Conveniently un-obfuscated files generate a all-zero key and don't need special treatment. The path name of each chunk is identical so it must be re-written and the header checksum must be re-calculated.
 
 After decompressing all files (quite a lot, it's split at 3k entries per file) from dat/teiln.dat (case insensitive), we get three kinds of files: one with all Vorname columns, one with all Nachname columns and one with the rest of the columns. Before 2000_Q1 files {0,3,6,...} were Nachname, {1,4,7,...} were Vorname and {2,5,8,...} were all the other tables. After 2000_Q1 (incl.) {0,3,6,...} were all other columns, while {1,4,7,...} were Nachname and {2,5,8,...} Vorname columns. The first char in each entry in the Nachname column contains a continuation flag, where "1" means single line entry, "3" is the first and "2" are the remaining lines in a continuation.
 
-If there is a dat/strassen.dat (case insensitive), the concatenated decompressed chunks are a list of all street names referenced in the street/hnr column.
+If there is a dat/strassen.dat (case insensitive), the concatenated decompressed chunks are a list of all street names referenced (hex, 0-based) in the street/hnr column.
 
 If there is a dat/karto.dat (case insensitive), the concatenated decompressed chunks are a sorted list of all zip/streetname(/hnr) combinations on that CD, each line finished by the geo coordinates of that address.
+
+version 4
+=========
+
+All strings are iso8859-1. All relevant information are in the files phonebook.db and streets.tl. If you need geo-coordinates to all your addresses, you'll also need the zip-streets-hn-geo.tl or zip-streets-hn.tl files, respectively. The files consist of zlib compressed chunks with streetname file just being a \0 separated list of streets, each referenced (dec, 0-based) from the street/hnr column. For the zip-streets*-geo.tl files, the concatenated decompressed chunks are a sorted list of all zip/streetname(/hnr) combinations on that CD, each line finished by the geo coordinates of that address (since they are sorted, you can do a binary search for each address in that file).
+
+The phonebook.db decompresses in a lot chunks with around 3000 entries sorted into 11 columns, the first being a binary flag byte and the latter: Nachname, Vorname, Namenszusatz+Addresszusatz, Verweise, Strassenindex_Hausnummer, Vorwahl, Postleitzahl, Ort, Rufnummer, Email+Webadresse respectively.
+
+The flag byte's lower nibble is 0 for a single line entry, 1 for the start of a continuation record and 2 for a trailing line of a continuation. The flag's top nibble bits are 0x80 set for a business record (as opposed to a natural person), 0x40 set, if the number must not be included in reverse query results and 0x20 used but unknown.
+