Proud Programmers Blog

One of the missing pieces in LotusScript has always been some kind of HttpReader, enabling us to communicate directly with the web. Lots of Notes-developers has therefore performed these tasks in Java within Notes/Domino. Java has excellent support for this, and luckily Notes/Domino host Java in a decent way.

Note that I don’t do anything new here, I simply combine stuff from various places. If you download and check out the code, you’ll see links to several sites where I got the original idea, snippet or code from.  Hopefully I haven’t forgotten any! One particular blog was very inspirational for me, Eric McCormic’s “Server REST with authentication”.

The whole trick is:

• A set of java classes with base class CoreConsumer. It contains most core logic, such as getContentCore and postContentCore. These methods contains the code that actually communicates with the REST Services. Subclasses JSONConsumer, XMLConsumer and HTMLConsumer contains more tuned methods for their respective content types, and also preset the content-type for you. At the time present, the JSONConsumer is the only one with a reasonable feature set. The XML and HTML consumers are only there to show the logic. Furture improvements may go along the lines of letting XMLConsumer have a powerful XPath query mechanism etc. JSONConsumer now has JSonPath-support and an internal XPath-like way of querying JSON.
  
• A LotusScript class wrapping the java class above via the LotusScript-to-Java bridge LS2J.
  
• Use the LotusScript-class in your own code as you see in the sample above.

How to use in your own code?

The sample database contain two script libraries, the “class:CWebContentConsumerJava” and “class:CWebContentConsumer”. The first script library is written in java, while the latter is written in LotusScript.

Copy these two script libraries to your own database, and you are ready to go.

Personally I like design inheritance, so I rather inherit the design from a central code-snippet database. Makes it much easier to maintain the code in one place! But that is of course up to you.

A litte heads-up on public REST services you can test on

In the following explanations I use REST services from some companies that kindly provide dummy services for us to use. Note that both the services can disappear or they can change the way they work, so please don’t focus too hard on the services themselves. If they disappear, simply look up other sample REST Services on the net.

A birds view on what you can do with the LotusScript class

Below you’ll see a lot of LotusScript code, simply using the script library class:CWebContentConsumer. Later I’ll point out some details about how the LS2J works and some details from the back-end java code.

You will also find the code presented below in two agents named “Test GET” and “Test POST” if you download the sample database. Note that I have skipped some of the ordinary variable definitions in the description below. The code in the sample database is fully working.

In the first screenshot you see how small the code can be.

Define the variable

Define your variable;

Dim json As New CJSONConsumer

Get the content from the REST Service

You can call a buch of methods before you get REST service’s content, but that is mostly if you need to control stuff like timeouts, what characterset you’d expect to receive and what not. Below I retrieve a list of countries from a REST Service;

strJson = json.GetJson("http://services.groupkt.com/country/get/all")

If the REST Service required authentication, the classes also support so-called basic authentication, with another method like this;

strJson = json.GetJsonWithAuth("http://services.groupkt.com/country/get/all", "Username", "Password")

Check errors

If everything goes well you have the complete JSON in the string variable! You can of course check if any errors occurred as well.

If json.GetLastErrorCode() <> 0 Then
    strLastError = json.GetLastErrorMessage()
End if

The classes support different kinds of errors, and you will get error messages from the back-end java classes, from the LS2J-bridge and from LotusScript itself. It might be a good thing to check on errors often

Inspect debugging information

The classes also support a simple debugging system, so you can track the different steps in the back-end java class. You may want to see extra debug information from the back-end java classes. Turn it on with SetJavaDebugLevel before other calls, and retrieve the result with GetJavaDebugInfo after other calls.

Call json.SetJavaDebugLevel(1) ' 0=Off (default), 1=On

The degug information can be retrieved with the GetJavaDebugInfo-method. All steps are delimited by a simple semi-colon (I said it was simple )

strValue = json.GetJavaDebugInfo()

Instruct the classes to behave your way before retrieving content

Remember to call most of these Set-methods before the Get-methods.

Timeout

Some REST Services may be slow to respond. By default the classes will wait 5 seconds. Perhaps you need to extend this time?

Call json.SetTimeout(16000) ' 16 seconds = 16000 ms

Character set

You may need to change the character set that the content is retrieved as. The default is UTF-8. Another often used character set are ISO-8859-1 

Call json.SetURLCharsetName("UTF-8")

Add Request Headers

Some REST Services needs additional co-called Request Headers to be set. This can be information such as customer-identification, tokens etc. You can add any number of additional request headers with AddRequestHeader, like this;

Call json.AddRequestHeader("MyID", "123")

Add URL Parameters

Some REST Services also needs URL parameters, which ends up in the URL itself. Instead of specifying them directly at URL-level, you can use the AddURLParameter to add any number of URL parameters.

Call json.AddURLParameter("CustomerID", "123")

Cleanup between calls if necessary

Note that you also can empty the request headers with EmptyRequestHeaders and ditto with the EmptyURLParameters.

What content-type do you want?

REST Services typically return their response in one of many so-called content types. You might receive JSON, XML, HTML, text or whatever. By default the classes CJSONConsumer sets the content type to "application/json", while CXMLConsumer sets "application/xml" and finally CHTMLConsumer use "text/html". You can set your own like this;

Call json.SetAccept("text/html")

Request Type

These classes has been designed to work primarily with GET and POST. The methods you see in the description typically reflects this with GetJson, GetJsonWithAuth, PostJson and PostJsonWithAuth. You can experiment with others by using;

Call json.SetRequestType("PUT")

User Agent

Many REST Services wants to know what kind of system requesting their services. This information is controlled by the request header User Agent. By default, these classes are very honest and report "no.vcode.webcontentconsumer/1.2". You can however set this to whatever you want, and thus spoof the REST Service;

Call json.SetUserAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11")

After retrieval methods

When the Get- or Post-methods has been called, you should have data within the bowls of these classes. Back to the initial REST Service-call;

strJson = json.GetJson("http://services.groupkt.com/country/get/all")

Below you see a screen shot of the service in a Chrome browser with a JSON viewer in place;

Each country has an element like this;

How to get values with JsonPath?

JsonPath Is a library that let you query the JSON almost like XPath. In other words, it is pretty easy to retrieve elements via queries.

If you are working with JSON, head over to the super resource  JSONPath Online Evaluator. Simply copy the JSON from the browser, and paste it into the online evaluator, and start querying. Below you see the c ountry-JSON and the instant query result!

First (marked by 1) I pasted the JSON. Then I entered the JsonPath-query, marked by 2. Instantly the result is shown at 3.

Simply copy the query and use it directly in your code, like this;

strValue = json.GetJsonPathValue("$.RestResponse.result[165].name")

You can of course grab elements and arrays and what not from the JSON. My classes also support that you can use a method prepareJsonPath with any other JSON, and use the same GetJsonPathValue-calls on your JSON.

How to get values by "json path"?

What?! Didn't you just show me JsonPath? Well, yes. JsonPath is all nice and good, but it isn't necessarily so easy to loop through JSON-arrays etc. Coming from LotusScript, I'd rather want a LotusScript-list, that I can loop through and find the same information. Therefore, the CJSONConsumer-class also have an "Array Maker", looping through the JSON after it has been retrieved by the Get- methods. Every JSON-element is given a "key" built up almost like XPath (hence "json path" …), making it possible to grab values like this;

strValue = json.GetJsonValueByKey("root\RestResponse\result\name:2:495")

As you see, the "path" consist of a "root" with the sub-elemenet levels following suit. If you have multiple, equal names, such as from the countries-JSON in this description, it is nice to know that every key also have information about both the level within the JSON file, and an unique record number within each level. So, from the key above, we can see that the "Norway"-name lives on level 3 (levels are 0-based), and have the unique record number 495. Now, this is not so easy to remember! If the JSON is simpler and not repeating equal elements, you can simply use;

strValue = json.GetJsonValueByKey("root\RestResponse\result\name")

… Which will retrieve the first name at the specified path.

Not good enough? You might get the internal structure yourself with the following calls;

iMax = json.GetJsonKeysAndValuesCount()
For iCur = 0 To iMax - 1
    strValue = json.GetJsonValueByIndex(iCur)
Next

How to POST instead?

The agent "Test POST" calls a REST Service and post information.

I have been using the RequestBin (https://requestb.in) which allows you to create a temporary URL and post up to 20 times to it.

These classes support posting with or without basic authentication, just like the Get-methods. When posting you often need to add additional HTTP request headers and perhaps even add URL parameters. When posting finishes, it often returns some kind of response in the form of response headers together with perhaps json or XML.

Note that all the sub classes (CJSONConsumer, CXMLConsumer and CHTMLConsumer, has heir own post-methods, in order to get everything right regarding content-types etc. This means that CJSONConsumer has a postJson and PostJsonWithAuth, CXMLConsumer has PostXML and PostXMLWithAuth and finally CHTMLConsumer has PostHTML and PostHTMLWithAuth.

You can, if you want to control everything yourself (using SetUserAgent, SetRequestType, SetURLCharsetName, SetTimeout etc), use the core-class CCoreConsumer's PostContent and PostContentWithAuth. This way you can control lots of stuff!

Below I show you how to call a REST service at the testing site https://requestb.in/ 

Remember, it it easy to *use* other rest services (in other words GET) and inspect the response etc, but it is somewhat harder to test a POST. This because it means that the  server needs to respond to arbitrary rest POSTS.

Services like RequestBin let you create a temporary "bin" meaning that you end up with your own bucket you can test to. Often these services also shows you want headers and payload the POST has posted.

In the sample below, I use the RequestBin URL https://requestb.in/ychjq7yc  NOTE THAT THIS MAY NOT EXIST WHEN YOU TRY THE CODE!!!!

You may use the URL https://requestb.in/ychjq7yc?inspect to see the responses!

The code below sets up a poster-variable, add some request headers, one URL parameter and post away;

Dim poster As New CJsonConsumer
Call poster.SetJavaDebugLevel(1)
Call poster.AddRequestHeader("MyHeader1", "Is there anybody out there?")
Call poster.AddRequestHeader("MyHeader2", "Hey ho, let's go!")
Call poster.AddURLParameter("CustomerID", "123")
strJson = poster.postContent("https://requestb.in/ychjq7yc")

The result can instantly be seen on RequestBin;

I have marked with red some of the request headers and parameters coming from my code.

Check the Response Headers

Both Get- and Post-methods also retrieve so-called Response Headers from the REST Services. In fact, that is also displayed by RequestBin, like this;

You can get all response headers with the following call;

strValue = poster.GetResponseHeaders("<My own separator>")

Or, you can get a single one, if you know the name.

strValue = poster.GetResponseHeader("Cookie")

By the way, the return code is always stored as "ReturnCode";

strValue = poster.GetResponseHeader("ReturnCode")

Use the back-end java JsonPath and "Array Maker" with your own JSON!

Behind the scenes, the methods that prepare the retrieved JSON-strings for use with both JsonPath and the "Array Maker", has also been made available to you.

For example, you stuff your own JSON like this;

strJson = |{ "returncode" : "ok" }|
bOk = poster.PrepareJsonPath(strJson)
If bOk Then
    strValue = poster.GetJsonPathValue("$.returncode")
End If
' Perhaps you want the Keys and Values instead? Kanena problema!
bOk = poster.PrepareJsonKeysAndValues(strJson)
If bOk Then
    strValue = poster.GetJsonKeysAndValues("<My sep>")
    strValue = poster.GetJsonValueByKey("root\returncode")
End If

Phew! Good luck folks, hope this helps somebody!

By the way; The downloadable sample database is here.