Part 2; Programming details on Icon and Images - Enumerating files
Tags: Lotus Notes Software DXL LotusScript
In the first article I introduced the Icon and Images database, in which I store more than 1.8 million icon files in a single database. This article dive into the programming details, showing how I enumerate the files to import.
In the first article I introduced the Icon and Images database, in which I store more than 1.8 million icon files in a single database. This article dive into the programming details, showing how I enumerate the files to import.
The Icon and Images
database primarily consist of two distinct passes, where the first is to
find out which files you want to import. I call this the enumeration
pass. The second pass is to create the icon imagery (the tables with
the icons, visible to the user) and attach the files themselves.
Pass 1 - Enumerating the files to import
In order to import icons and images, I need to know where they are stored. I also need to know a lot about the directory structure the icons have. As illustrated in the previous article, the VirtualLNK icon collection has 8 different directory structure types. I need to know the structure in order to grab icon attributes like state, shadow, and size for each icon. Lets take a look at an example. The Network_V2 zip file consists of yet a bunch of zip files, like this;
When you dive into a sub-zip file you find that this library has a structure like this;
G:\Temp\Network_V2\Network_V2\PNG\PNG\Regular\No Shadow\48x48
Note that this directory structure not necessarily is valid for other zip files!! In order to control this, I have identified the different directory structures available, and rather created a Database Configuration document, which let me specify parts of the directory mapped to the appurtenant directory structure, like this;
In the above screenshot I can tell that any icon file which contain the Network_V2 anywhere in its path (like G:\Temp\Network_V2\Network_V2\PNG\PNG\Regular\No Shadow\48x48\Antenna.png,.,) will have the directory structure equal to 2. The available directory structures I have are;
To best understand the table above, start reading it from right to left. When I state that the ICO level is either at the same level or one less level, it indicates that the ICON files doesn't have all the levels of paths as the other files. This makes sense since the icon files often have all available sizes baked into a single ico or icns file. Thus they don't need the <size> part.
The different path-part names in the table above are;
Also please note that there are several ways of unpacking such zip files. I have chosen to unpack into separate folders - always. This is extremely convenient with WinRAR, which I use as my primary ZIP tool.
The pass 1 dialog box
The above dialog box let the user specify the vendor, which directory structure type he or she is about to import, and the base import folder. Note that the dialog box above has selected AUTOMATIC for the directory structure type . This means that we use the directory-to-type map described earlier in this article. It makes it much much easier to import huge numbers of icons.
If I dive into the LotusScript code behind the pass 1 agent, I the main logic is pretty simple. First of all I use a recursive function to walk the import folder. This function will basically enumerate all the files within the current level of directory structure. It will also remember all the directories within the current directory structure. For each sub-directory it will call itself (this is where the recursive part comes in...) and do the same enumeration again for the next level in the directory structure. This way it walks its way through all directories beneath the initial base directory.
The code looks like this;
Function EnumerateImportFiles(session As NotesSession, _
db As NotesDatabase, _
view As NotesView, _
pstrPath As String, _
iRecursive As Integer) As Long
EnumerateImportFiles = 0 ' Default; No files found
filename = Dir$(pstrPath & WILDCARD_ALL_FILES, 16)
Do While (Not filename = "")
If filename <> "." And filename <> ".." Then
strFullFile = CreateFullPath(pstrPath, filename)
lAttributes = Getfileattr(strFullFile)
' Do we have a directory ? If so, remember that 'till later!
If lAttributes And 16 Then
iNbrDirectories = iNbrDirectories +1
listDirectories( iNbrDirectories) = strFullFile
Else
' If iBasePathType is -1 it means that we have
' automatic base path type matching with the map
' specified in the Database Configuration document
If g_iBasePathTypeFromUI = -1 Then
iBasePathType = GetBasePathTypeFromMap(session, _
db, strFullFile)
Else
iBasePathType = g_iBasePathTypeFromUI
End If
' The next function split the file path into their
' respective parts. Note how the first parameter
' define how to analyze the path according to directory
' layout in VirtualLNK
Call SplitPathIntoPathParts(iBasePathType, strFullFile, _
strFPath, strFName, strFExt, strImageType, strState, _
strShadow, strSize, strLibraryVersion, strLibrary)
If (iBasePathType <> -1 And (strFExt = "JPG" Or strFExt = "GIF"_
Or strFExt = "PNG" Or strFExt = "ICO" Or strFExt = "ICNS" _
Or strFExt = "BMP")) Then
lFileCount = lFileCount + 1
Print "PASS 1: Enumerating file # " & Cstr(lFileCount + _
g_lNbrFilesToImport) & " in library " & strLibrary
'##############################################
' The REAL PEPPER GOES ON HERE
'##############################################
End If ' end If iPathNbrOfParts > 5
End If
End If
filename = Dir$
Loop
' If directories was found -and- we want to search sub dirs, then do that now
If iRecursive = 1 And iNbrDirectories > 0 Then
Forall strDirectory In listDirectories
lAttributes = Getfileattr(strDirectory)
If lAttributes And 16 Then
If Right(strDirectory,1) <> "\" Then strDirectory = strDirectory _
& "\"
EnumerateImportFiles = EnumerateImportFiles + _
EnumerateImportFiles(session, db, view, strDirectory, iRecursive)
End If
End Forall
End If
End Function
Some notes about the code above.
As you see, I use the LotusScript Dir$ function to list the files and directories. To make it easier for myself, I store all directories in a temporary list, and all files are processed right away, To determine whether a file is a directory or a file, I use the GetFileAttr-function.
In order to understand what VirtualLNK directory structure type we are working with, I have a function named GetBasePathTypeFromMap, which return the directory structure type number for the file. This is used as input parameter to the function SplitPathIntoPathParts, which breaks apart the path for each file, and return the path parts. This is where it so important to really have control of the directory structure! If I recognize the file as a image file I want to process, then the code ends up at The REAL PEPPER GOES ON HERE. In the real code I basically check whether I have imported the file before or not. If it is a new file, I will create a new formIcon document in the database.
How do I understand the uniqueness of an icon? By combining the Library Version with the file name part of the icon, I have the key of the icon. This means that all variants of for example the Accountant icon will be collected within the very same Notes document. By also including the library version-part I can have multiple icons with the same name across multiple libraries.
When pass 1 is finished, I have a bunch of unique icon documents, and each document contains a Files -field containing the full path to every icon file. The number of files are also seen in the main view;
The next article
The next article will dive into how I create the icon imagery in the documents. That will be a heavy-duty DXL article.
Pass 1 - Enumerating the files to import
In order to import icons and images, I need to know where they are stored. I also need to know a lot about the directory structure the icons have. As illustrated in the previous article, the VirtualLNK icon collection has 8 different directory structure types. I need to know the structure in order to grab icon attributes like state, shadow, and size for each icon. Lets take a look at an example. The Network_V2 zip file consists of yet a bunch of zip files, like this;
When you dive into a sub-zip file you find that this library has a structure like this;
G:\Temp\Network_V2\Network_V2\PNG\PNG\Regular\No Shadow\48x48
Note that this directory structure not necessarily is valid for other zip files!! In order to control this, I have identified the different directory structures available, and rather created a Database Configuration document, which let me specify parts of the directory mapped to the appurtenant directory structure, like this;
In the above screenshot I can tell that any icon file which contain the Network_V2 anywhere in its path (like G:\Temp\Network_V2\Network_V2\PNG\PNG\Regular\No Shadow\48x48\Antenna.png,.,) will have the directory structure equal to 2. The available directory structures I have are;
| Type | Path |
|
0
| <Base dir>\<Library>\<Library Version>\<Shadow>\<Shadow>\<Image Format>\<State>\<Size> |
|
1
| <Base dir>\<Library>\<Library Version>\<Image Format>\<Image Format>\<State>\<Size>\<Modificator> (ICO same level) |
|
2
| <Base dir>\<Library>\<Library Version>\<Image Format>\<Image Format>\<State>\<Shadow>\<Size> (ICO same level) |
|
3
| <Base dir>\<Library>\<Library Version>\<Image Format>\<Image Format>\<State>\<Size> (ICO same level) |
|
4
| <Base dir>\<Library>\<Library Version>\<Image Format>\<Image Format>\<State>\<Size>\<Modificator> (ICO one less level) |
|
5
| <Base dir>\<Library>\<Library Version>\<Image Format>\<Image Format>\<State>\<Size>\<Shadow> |
|
6
| <Base dir>\<Library>\<Library Version>\<Image Format>\<Image Format>\<State>\<Size> (ICO one less level) |
|
7
| <Base dir>\<Library>\<Library Version>\<Image Format>\<Size> (ICO one less level) |
To best understand the table above, start reading it from right to left. When I state that the ICO level is either at the same level or one less level, it indicates that the ICON files doesn't have all the levels of paths as the other files. This makes sense since the icon files often have all available sizes baked into a single ico or icns file. Thus they don't need the <size> part.
The different path-part names in the table above are;
| Path Part | Description | |||||||
| <Size> | The
size of the images, typically "256x256", "128x128",
"16x16". Below you see a sample of the same icon ranging from
"128x128" and down to "16x16";
| |||||||
| <State> | The
state of the image, typically "Hot", "Disabled" or
"Normal".Very often icons used for toolbars in applications has
such states. Below you see the same icon at size "48x48" in its
three different states, Normal, Disabled and Hot;
  
| |||||||
| <Modificator> | Several
VitualLNK icon collections has a special organization where they have a
so-called base image. This is the pure and clean base image. Then
they have modificators "on top" of the base image, such as for
"Add", "Delete", "Check" etc. Below you see
a small sample of this;
The base image; 
The base image with the "add" modificator; 
The base image with the "chat" modificator;  The purpose of the modificators is to make it easy to have variations of the same icon. | |||||||
| <Shadow> | The
shadow of the icon indicates whether the icon has an shadow or not, like
this;
No shadow; 
Dark shadow; 
Light shadow; 
| |||||||
| <Image Format> | The image format is first and foremost it's filetype. Be aware though, that an image type like BMP may occur in two different modes, such as "BMP" and "BMP_Mask". The same goes for "ICO" files which often comes as "ICO_8Bit" and "ICO_24bit". The way I unpack the directores, I often end up with two consequent image formats path-parts, where the right-most always are the image type (BMP, PNG, ICO, GIF etc) and the next right-most are the image format as described. | |||||||
| <Library Version> | The library version is typically the version path-path like "Network_V2" or "Business V3" | |||||||
| <Library> | The library is the main library name, like "Network" or "Business" | |||||||
| <Base dir> | The base directory is the left-most base directory, such as "G:\Temp" |
Also please note that there are several ways of unpacking such zip files. I have chosen to unpack into separate folders - always. This is extremely convenient with WinRAR, which I use as my primary ZIP tool.
The pass 1 dialog box
The above dialog box let the user specify the vendor, which directory structure type he or she is about to import, and the base import folder. Note that the dialog box above has selected AUTOMATIC for the directory structure type . This means that we use the directory-to-type map described earlier in this article. It makes it much much easier to import huge numbers of icons.
If I dive into the LotusScript code behind the pass 1 agent, I the main logic is pretty simple. First of all I use a recursive function to walk the import folder. This function will basically enumerate all the files within the current level of directory structure. It will also remember all the directories within the current directory structure. For each sub-directory it will call itself (this is where the recursive part comes in...) and do the same enumeration again for the next level in the directory structure. This way it walks its way through all directories beneath the initial base directory.
The code looks like this;
Function EnumerateImportFiles(session As NotesSession, _
db As NotesDatabase, _
view As NotesView, _
pstrPath As String, _
iRecursive As Integer) As Long
EnumerateImportFiles = 0 ' Default; No files found
filename = Dir$(pstrPath & WILDCARD_ALL_FILES, 16)
Do While (Not filename = "")
If filename <> "." And filename <> ".." Then
strFullFile = CreateFullPath(pstrPath, filename)
lAttributes = Getfileattr(strFullFile)
' Do we have a directory ? If so, remember that 'till later!
If lAttributes And 16 Then
iNbrDirectories = iNbrDirectories +1
listDirectories( iNbrDirectories) = strFullFile
Else
' If iBasePathType is -1 it means that we have
' automatic base path type matching with the map
' specified in the Database Configuration document
If g_iBasePathTypeFromUI = -1 Then
iBasePathType = GetBasePathTypeFromMap(session, _
db, strFullFile)
Else
iBasePathType = g_iBasePathTypeFromUI
End If
' The next function split the file path into their
' respective parts. Note how the first parameter
' define how to analyze the path according to directory
' layout in VirtualLNK
Call SplitPathIntoPathParts(iBasePathType, strFullFile, _
strFPath, strFName, strFExt, strImageType, strState, _
strShadow, strSize, strLibraryVersion, strLibrary)
If (iBasePathType <> -1 And (strFExt = "JPG" Or strFExt = "GIF"_
Or strFExt = "PNG" Or strFExt = "ICO" Or strFExt = "ICNS" _
Or strFExt = "BMP")) Then
lFileCount = lFileCount + 1
Print "PASS 1: Enumerating file # " & Cstr(lFileCount + _
g_lNbrFilesToImport) & " in library " & strLibrary
'##############################################
' The REAL PEPPER GOES ON HERE
'##############################################
End If ' end If iPathNbrOfParts > 5
End If
End If
filename = Dir$
Loop
' If directories was found -and- we want to search sub dirs, then do that now
If iRecursive = 1 And iNbrDirectories > 0 Then
Forall strDirectory In listDirectories
lAttributes = Getfileattr(strDirectory)
If lAttributes And 16 Then
If Right(strDirectory,1) <> "\" Then strDirectory = strDirectory _
& "\"
EnumerateImportFiles = EnumerateImportFiles + _
EnumerateImportFiles(session, db, view, strDirectory, iRecursive)
End If
End Forall
End If
End Function
Some notes about the code above.
As you see, I use the LotusScript Dir$ function to list the files and directories. To make it easier for myself, I store all directories in a temporary list, and all files are processed right away, To determine whether a file is a directory or a file, I use the GetFileAttr-function.
In order to understand what VirtualLNK directory structure type we are working with, I have a function named GetBasePathTypeFromMap, which return the directory structure type number for the file. This is used as input parameter to the function SplitPathIntoPathParts, which breaks apart the path for each file, and return the path parts. This is where it so important to really have control of the directory structure! If I recognize the file as a image file I want to process, then the code ends up at The REAL PEPPER GOES ON HERE. In the real code I basically check whether I have imported the file before or not. If it is a new file, I will create a new formIcon document in the database.
How do I understand the uniqueness of an icon? By combining the Library Version with the file name part of the icon, I have the key of the icon. This means that all variants of for example the Accountant icon will be collected within the very same Notes document. By also including the library version-part I can have multiple icons with the same name across multiple libraries.
When pass 1 is finished, I have a bunch of unique icon documents, and each document contains a Files -field containing the full path to every icon file. The number of files are also seen in the main view;
The next article
The next article will dive into how I create the icon imagery in the documents. That will be a heavy-duty DXL article.
| Permalink | 
Comments
Posted by Art At 00:12:46 On 09.05.2008 | - Website - |