Proud Programmers Blog :: Unexpected results reading a plain text file with NotesStream in LotusScript (or when strings contain MORE than meet the eye)

Tags: LotusScript

This is a little LotusScript experiment with a plain text file and a LotusScript agent using NotesStream. After reading the text from the file I try to use the string to locate a document with a corresponding key in a NotesView. While everything looks fine in the debugger, something is not correct and the lookups fail. This article has some suggestions for solutions at the end but they are not state-of-the-art! Perhaps you know other ways?

First off, I have a super-simple plain text file with a single number in it;

SNAGHTML6c68e14

Then I have a super-simple LotusScript agent reading this text file, and trying to locate a Notes document with the same number from a NotesView.

SNAGHTML6c794b7

The documents have all their numbers stored as text, like this;

SNAGHTML6c802d2

Why stored as text? Because this is how I found the (to me) abnormality in the first place Smile And why NotesStream? Because code with the concept herein also runs on an iSeries, where it works fine …

The LotusScript-code looks like this;

The code uses a NotesView (1) and a NotesStream (2) to open the file and read out the contents with the ReadText-method (3). Finally the code tries to locate the document in the NotesView first by a hardcoded key (4), and then by the value retrieved via NotesStream (5). In a moment you will see how the first works and the second not.

I use the LotusScript debugger and step through the code to see the contents of the variables. First breakpoint is just before the first GetDocumentByKey-call above, when I use the hardcoded key “123456” (marked as 1 below);

As you see from the variable panel, the NotesDocument “doc” hasn’t got any value at all yet (2), and you see that the strLine-variable has the value “123456” – or so it seems ….

I step the code one line down, just to see that the first GetDocumentByKey actually does find the document with the key “123456”;

So we now know that the document is “findable” Smile

I am now ready to step the code one line more, to process the second GetDocumentByKey (marked as 1 below), now with the content from the strLine-variable, containing “123456” (2);

…but wait! A closer look at the strLine in the variables pane (3) shows that strLine doesn’t have a closing apostrophe! Something is different here and the result from the NotesStream’s ReadText is obviously containing something more other than just the text.

So, when I run the last line of code, it won’t find the document …

What on earth is going on here? Obviously it it something with the characterset or NotesStream-way of handling things.

My first attempt was to strip off whatever strLine is holding at the end of the string, along these lines;

The code above “leftifies” the content of strLine by the length of the string itself. While this code seems to “repair” the strLine string variable …

… it doesn’t find the document…

In my case I had to do far too elaborate stuff like this, where I first convert the number to a double (just so I can store huge numbers ..) and then format it back to a string with the LotusScript’s Format() function;

And now the code works!! As you see in the variables pane, both strLine and strKey look exactly the same, but now the content of strKey is obviously on the correct format …

Another way of getting the code to work is to getting rid of all 0’s, CR and LFs from the string, such as;

strLine = Replace(strLine, Chr(0), "")
strLine = Replace(strLine, Chr(10), "")
strLine = Replace(strLine, Chr(13), "")

This also works and the strLine now contain a valid string without something-at-the-end!!

But, there must be another way?! Some parameter to set on the NotesStream-object?

PS! I have also tried to use different character sets when opening the file such as UTF-8 and US-ACIII. Additionally I have tried to use other line-end flags in ReadText such as EOL_CR and EOL_CRLF. Same results unfortunately.

I also found this post, but that didn’t work for me.

Comments

1 - I strongly suspect it has to do with the encoding of the file. I had problems with using NotesStream on the iSeries, where NotesStream.Open with a charset specified will NOT set the correct CCSID on the created file in the IFS which later leads to problems.
SPR CPAPABUR4M and APAR LO89645 where created from the PMR. I suggest you open a PMR with IBM referencing SPR CPAPABUR4M and APAR LO89645 so your problem adds weight to the problem and the chance of getting a fix increases.

My problem was that on IBM i, the NotesStream.Open call with charset "Windows-1252" and then writing text to it created a file in IFS which contained text in the Windows-1252 character set, but was marked with CCSID 37 (EBCDIC-US) instead of CCSID 1252 (Windows-1252).

If you then use software to process this file which uses the CCSID property, like native iSeries programs or FTP, the conversion renders the contents into garbage. FTP can be worked around by using the BIN command to disable character set conversion.
Native programs can be made to work by setting the charset in the NotesStream.Open call the same as the IFS default.

Another workaround is using the Open statement from LS instead of the NotesStream.Open call.
The Open statement WILL honor the charset parameter.

Note that determining the CCSID of a file can be done by using System i Navigator to browse the Integrated File System, browse to the problem file, and looking at the Storage tab of the File Properties to see which CCSID is set on the file.

hope can use this info!

Posted by Lars Berntrop-Bos At 14:06:13 On 05.07.2017 | - Website - |

2 - It's not via NotesStream, but I've used LineImport, and I think it works on iSeries : { Link }

I wonder if the NotesStream is getting a line break and trying to use that in the key?

Cheers,
Brian

Posted by Brian Moore At 14:36:23 On 05.07.2017 | - Website - |

3 - @Lars, thanks for your answer! The file is a plain Windows file with no encoding at all. Looking at the file in a hex editor, reveal the following hex-sequence; 31 32 33 34 35 36 0D 0A. In other words, the ASCII values of 123456 with a CRLF at the end. My text-file wrapper class on iSeries actually open the file as binary in order to bypass any CCSID-related problems. For some reason this doesn't work *in the same way* on Windows. I guess I could use standard LS-Open and force binary on that too to achieve the same result.

Posted by Robert Ibsen Voith At 11:03:49 On 10.07.2017 | - Website - |

4 - @Brian, thanks for your answer and your link! It is definately a newline in the end of the file! As you see from my other answer, the hex sequence of the file is; 31 32 33 34 35 35 0D 0A. The main problem - in my opinion, is that NotesStream doesn't reveal this properly in the debugger and thus it is very easy to overlook. However, forcing the code as specified in the blog-post will fix the problem. Perhaps there is other ways Emoticon