MLIST - JSON
#22265
06 Nov 16 10:43 PM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
Here's another follow-up on our MLIST exercises from the Conference: a set of routines to parse a JSON document into an MLIST. The representation scheme chosen here is that each MLIST element contains one of the following: [] (sublist contains array)
{} (sublist contains object)
string (string scalar)
scalar (number, true, false, null)
"name":"string" (name:value pair)
"name":scalar (name:value pair)
"name":{} (value part of pair is object in sublist)
"name":[] (value part of pair is array in sublist) So for the following JSON document (a common sample found on the Internet)... { "store": {
"book": [
{ "category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{ "category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
},
{ "category": "fiction",
"author": "Herman Melville",
"title": "Moby Dick",
"isbn": "0-553-21311-3",
"price": 8.99
},
{ "category": "fiction",
"author": "J. R. R. Tolkien",
"title": "The Lord of the Rings",
"isbn": "0-395-19395-8",
"price": 22.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
} ... the MLIST representation would appear as shown below (using the Mlist'Multi'Level'Display() routine from mlistdemo.bsi; note that the ">" characters are just to indicate the nesting level) ... > {}
> "store":{}
> "book":[]
> {}
> "category":"reference"
> "author":"Nigel Rees"
> "title":"Sayings of the Century"
> "price":8.95
> {}
> "category":"fiction"
> "author":"Evelyn Waugh"
> "title":"Sword of Honour"
> "price":12.99
> {}
> "category":"fiction"
> "author":"Herman Melville"
> "title":"Moby Dick"
> "isbn":"0-553-21311-3"
> "price":8.99
> {}
> "category":"fiction"
> "author":"Evelyn Waugh"
> "title":"Sword of Honour"
> "price":12.99
> {}
> "category":"fiction"
> "author":"Herman Melville"
> "title":"Moby Dick"
> "isbn":"0-553-21311-3"
> "price":8.99
> {}
> "category":"fiction"
> "author":"J. R. R. Tolkien"
> "title":"The Lord of the Rings"
> "isbn":"0-395-19395-8"
> "price":22.99
> "bicycle":{}
> "color":"red"
> "price":19.95 In other words, the JSON document was mapped onto an MLIST 5 levels deep: - The top level contains just one element - {} - i.e. a single JSON object. - Attached to that parent node is a sublist consisting of of a single name:value pair, where the name is "store" and the value is another object, stored in its sublist. - The contents of the store object at the next level down consists of two name:value pairs. The first is a "book", whose value is an array of books; the second is "bicycle", whose value is another object. - Below "books":[] we have an array of objects, each of which has a sublist containing a series of name:value pairs representing attributes of the books. And so on. Representing the JSON document in an MLIST like this may not seem like a great simplification over the original text format, particularly in this case where the JSON was already nicely formatted for human reading. (In general that isn't necessarily the case.) The advantage the MLIST structure is that it because normalized and thus more amenable to to the use of generic routines (possibly with custom callback functions) to perform routine operations such as walking the tree, searching, formatting, etc. The Mlist'Multi'Level'Display() display routine is a simple example of such a generic routine which is able to format with standard indentation any such document. Perhaps in a subsequent post we can consider some more interesting examples. In the meantime, I've posted the MLISTJSON.BSI module containing the parsing routines, in the EXLIB Repository (908053) along with a sample test program MLISTJSON1.BP, for anyone motivated to play with it. Although there are a couple of helper functions to read text from the file and grab the next token, the bulk of the logic translating between the JSON and MLIST representations is all in this relatively simple (but recursive) routine: !---------------------------------------------------------------------
!Function:
! parse a JSON array or object, up to final "]" or "}"
!Params:
! $json() (mlist) [ref] - parsed JSON data stored here (see notes at top)
! endtoken$ (s) [in] - ending token to look for ("]" or "}")
!Returns:
! >= 0 for success
! < 0 for error
!Module vars:
! chin, workbuf$
!Notes:
!---------------------------------------------------------------------
Function Fn'MLJ'Parse($json() as mlist(varstr), endtoken$ as s2:inputonly) as i4
map1 locals
map2 token$,s,0
map2 status,i,4
map2 bdone,BOOLEAN
map2 subendtoken$,s,2
level += 1
do
! get next [,{,],},name:{,name:[, or scalar from file
token$ = fn'mlj'next'token$()
switch token$
case "[" ! parse a new array
case "{" ! parse an object
$json(.pushback) = token$
subendtoken$ = ifelse$(token$="[","]","}")
status = Fn'MLJ'Parse($json(.back).sublist,subendtoken$)
if status >= 0 then
$json(.back) += subendtoken$
else
$json(.back) += " !!Error: missing terminator: "+subendtoken$
endif
exit
case "]"
case "}"
status = ifelse(token$=endtoken$,0,-1)
bdone = .TRUE
exit
case "" ! no more data (error?)
bdone = .TRUE
status = -1
exit
default
switch token$[-1,-1]
case "{" ! name:{
case "[" ! name:[
$json(.pushback) = token$
subendtoken$ = ifelse$(token$[-1,-1]="[","]","}")
status = Fn'MLJ'Parse($json(.back).sublist,subendtoken$)
if status >= 0 then
$json(.back) += subendtoken$
else
$json(.back) += " !!Error: missing terminator: "+subendtoken$
endif
exit
default
! add an name:value, or scalar element to our list
$json(.pushback) = token$
exit
endswitch
exit
endswitch
loop until bdone
if status < 0 then
Fn'MLJ'Parse = -1
endif
level -= 1
EndFunction
|
|
|
Re: MLIST - JSON
#22266
07 Nov 16 01:33 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Nice! and im sure will be useful
|
|
|
Re: MLIST - JSON
#22267
07 Nov 16 06:55 PM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
FYI, I updated the MLISTJSON.BSI to add a serialization routine (to write the JSON data stored as an MLIST back to a file or string variable). And also the test program MLISTJSON1.BP accordingly.
|
|
|
Re: MLIST - JSON
#22268
08 Nov 16 06:24 AM
|
Joined: Jun 2001
Posts: 153
OmniLedger - Tom Reynolds
Member
|
Member
Joined: Jun 2001
Posts: 153 |
This is exactly the sort if thing I have in mind for using it for. Simplifying XML and JSON calls into common handlers would save a lot of time.
I've already done this by writing some standard functions that flatten an xml file and allow easy access to it, but I'm almost certain MLIST would give me better performance and memory usage over my current implementation.
|
|
|
Re: MLIST - JSON
#22269
08 Nov 16 08:42 AM
|
Joined: Jun 2001
Posts: 3,406
Jorge Tavares - UmZero
Member
|
Member
Joined: Jun 2001
Posts: 3,406 |
Oh boy! it seems that this came right on time to use in a new webservice fom where I have to collect JSON data
Jorge Tavares
UmZero - SoftwareHouse Brasil/Portugal
|
|
|
Re: MLIST - JSON
#22270
08 Nov 16 11:16 AM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
Well it does certainly seem that: a) MLIST more closely matches the natural data structure of JSON and XML than other data types we have at our disposal. b) It would be nice to have a common set of utility functions available for the most common tasks, rather than everyone writing their own from scratch. But beyond the obvious tasks (Load, Tree Walk, Unload/Serialize), it's not clear just what other kinds of functions would be generally useful. But I have some thoughts about it... 1. For large JSON documents, it might be nice to create an index to the values, which could be in the form of an ORDMAP whose keys were whole or partial values (e.g., for the store example above, things like "Tolkien", "Rings", "Moby", "Dick", "red", etc.), and whose values were ....? Some kind of pointer to the location in the structure? The problem there is that there is no mechanism to store a reference/pointer to an arbitrary position in an MLIST. And even if you could, it's not always clear what you would want to point to. For example, if you searched for "red", you'd probably want the result to indicate the context, which in that case would include the parent (bicycle) as well as the sibling (the price). If searching for "Tolkien", we might want to get an array of book objects. 2. One scheme for indexing/flattening a tree structure like JSON or XML is to use an XPATH-like notation. (XPATH is a formal standard for XML, but as far as I know, there isn't yet a formal equivalent for JSON, although we could probably come up with a workable one.) In fact, attendees of the 2014 Conference may remember this routine: xcall JSON,1,handle,status,jsonstring$,$jsonmap() which takes a JSON document in the form of a string and creates an XPATH-like ORDMAP from it. For the store document above, it would look like this: /store/bicycle/color = red
/store/bicycle/price = red
/store/book[1]/author = Nigel Rees
/store/book[1]/category = reference
/store/book[1]/price = 8.95
/store/book[1]/title = Sayings of the Century
/store/book[2]/author = Evelyn Waugh
/store/book[2]/category = fiction
/store/book[2]/price = 12.99
/store/book[2]/title = Sword of Honour
/store/book[3]/author = Herman Melville
/store/book[3]/category = fiction
/store/book[3]/isbn = 0-553-21311-3
/store/book[3]/price = 8.99
/store/book[3]/title = Moby Dick
/store/book[4]/author = J. R. R. Tolkien
/store/book[4]/category = fiction
/store/book[4]/isbn = 0-395-19395-8
/store/book[4]/price = 22.99
/store/book[4]/title = The Lord of the Rings The JSON.SBR routine was never really released (although it's in the ashw32 executable) because at the time it wasn't clear that using an embedded XCALL to operate on JSON was a good idea. (Too difficult to enhance incrementally.) Now we could easily implement that same function in A-Shell/BASIC. For example, from the MLIST representation of the JSON document we might generate two ORDMAPs - one like the version just shown above, and the other being being the reverse, i.e. the index, mentioned before that. It may be a bit overkill, but that would allow you to browse the values and then map from, say "Moby Dick" to /store/book[3], and then from "/store/book[3]" you could easily iterate through all the child items (i.e. the name:value attribute pairs of that book). 3. One problem with the ORDMAP representation is that it comes out in ASCII order. That's ok for the name:value pairs whose physical order is not important. But technically, the elements in the JSON array structure are ordered, i.e. the physical order might be important. So it might be useful to also have a function that takes an XPATH-like location reference and then finds that location in the MLIST and perhaps iterates its children, or returns a new MLIST just consisting of the children. 4. Many discussions of JSON (including the introduction at http://www.json.org/ ) suggest that JSON objects (lists of name:value pairs) be represented by a some kind of associative array, like ORDMAP, while JSON arrays (ordered lists of values) be represented by some kind of list (i.e. MLIST). But since the elements of arrays, as well as the values of name:value pairs, can themselves be arrays or objects, the suggested bifurcation could lead to quite an complex hierarchical network of individual such arrays and lists. Decomposing an entire document into a vast assemblage of arrays and lists all at once seems of dubious value, but it may make sense to apply that technique to small subsets of the data that can be first identified by some kind of search. In other words, there may be room for functions that, given some kind of focus or scope (possibly resulting from a set of search criteria), return an ORDMAP and/or an MLIST (and/or possibly other data structures) representing just that subset of data. 5. For documents of a reasonable size (let's say < 500K), or which aren't going to be searched/manipulated too heavily, it may be that just leaving it in the MLIST format and writing routines to search/manipulate it directly, makes more sense than getting fancy with ORDMAPs, indexes, other another such "overlays". For example, if all we want to do is get a list of the titles and prices for books written by Tolkien, a single walk through the MLIST, using a very generic routine with a custom callback helper routine to pull out the items of interest, would probably be the best approach. The main difficulty here is the problem of obtaining the larger context -- parents, grandparents, siblings, cousins, etc. -- once a particular item is located. The various mlistxml* programs and bsi modules in 908053 of the EXLIB which were reviewed at the Conference offer up a few possibilities for how to tackle that challenge, but they could no doubt benefit from some review/refinement and/or additional approaches/ideas. I'll keep tinkering along these lines, time permitting, as long as it seems productive. But all suggestions and contributions will certainly be appreciated (including telling me when to stop! :rolleyes: )
|
|
|
Re: MLIST - JSON
#22271
08 Feb 17 09:15 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
In the next few days im about to write a new little routine that looks up a PostCode (zip) and returns a Json from a URL. Its based on THIS format. Im sadly handicapped as this is for AIX Customer running 6.0.1256.2 so im unable to update them and use all the latest bells and whistles like MLIST etc. What I was thinking of doing is writing a JON to CSV converter (or on those lines as it could be useful in future) in this case I know the fields / format im receiving, Anyway just wondering if half the parsing side is done already reading the file structure and I can plug in the output.. :rolleyes: Heres a Json example im getting back.. Anyway thought i'll ask before I plough on...
|
|
|
Re: MLIST - JSON
#22272
08 Feb 17 09:17 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Oh the file returned is one long line, is there any line size limit on INPUT LINE etc now days.
|
|
|
Re: MLIST - JSON
#22273
08 Feb 17 11:33 AM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
No, I don't think there is any absolute limit. Here's a little test program that allows you to try various sizes. It works ok with a line 5 million bytes long, but if that's not enough, you can try another value. (At some point you may run out of available memory, but that's a different issue.) ! test limits of input line into s,0
map1 n,f
map1 pline,s,0
map1 i,f
input "Enter # bytes to test: ",n
! Create file with one long line...
open #1, "inputs0.lst", output
for i = 1 to n
? #1, "a";
next i
close #1
! now try to read it
open #1, "inputs0.lst", input
input line #1, pline
close #1
? "len(pline) = ";len(pline)
? pline[1,70];"..."
end As for what to do about AIX, there's no plan to support the extended container structures (ordmap, mlist) there. But if you really wanted to use that approach, you could create an SBX to execute on the ATE side. You'd probably want to file transfer the data back and forth, but maybe that wouldn't be unreasonable, depending on the usage pattern. The XCALL JSON (see the post above) is implemented in plain C and thus could easily be ported to AIX, but as it currently stands, the only useful function of the routine is to translate the JSON document into an ordmap (which won't be possible under AIX). But it probably wouldn't be that difficult to translate into a similar structure that was then just output as a file, although I'm not sure how much help that would be. JSON to CSV sounds nice, but obviously would only make sense for a limited set of JSON structures.
|
|
|
Re: MLIST - JSON
#22274
08 Feb 17 02:43 PM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Oooo I like your thinking on creating an SBX and executed on the ATE side, I guess I can call xcall Json then too?
|
|
|
Re: MLIST - JSON
#22275
08 Feb 17 04:39 PM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
|
|
|
Re: MLIST - JSON
#22276
09 Feb 17 01:00 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
|
|
|
Re: MLIST - JSON
#22277
09 Feb 17 08:59 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Ok I have now a new little SBX that converts THIS JSON to THIS CSV I've also got the SBX to transfer locally for ATE to execute, but having mix and uneventfully results executing it locally, whats the best way? is it using PRINT tab(-10, AG_XFUNC), XFUNC() or MX_AGWRAPPER?
|
|
|
Re: MLIST - JSON
#22278
09 Feb 17 09:35 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Update - I think PRINT TAB(-10, AG_XFUNC) is launching it locally fine as a MIAMEX,MX_ASHLOG shows a trace in the local ashlog.log - but im not getting any parameters passed along..
|
|
|
Re: MLIST - JSON
#22279
09 Feb 17 09:47 AM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
MX_AGWRAPPER is just a wrapper around PRINT TAB(-10,AG_XFUNC{S}) followed by an INPUT to retrieve the result. So MX_AGWRAPPER is cleaner (eliminates concerns about input modes). But either assumes that the SBX is already on the ATE side. Perhaps an even cleaner approach would be to just XCALL your SBX and let the copy of the SBX which is on the server replicate itself (if necessary) to the client and then redirect the call. There is a handy wrapper function for this in the SOSLIB: Fn\'ATE\'SBX() . That version is designed only to return a numeric value, but you could easily create a Fn'ATE'SBX$() variation of it to return a string value (perhaps the entire CSV output?). Or, maybe the server side of the SBX, after invoking the client side to do the conversion, would then retrieve the file using AG_FTP or ATEGFK . Offhand I'm not sure if there is a practical limit on the size of the string returned by AG_XFUNCS, but I am sure there's no limit on the size of a file retrieved by any of the file transfer methods, so depending on the expected range of sizes of the CSV output, it might be more robust to use the file transfer approach. I see you've emailed me the source code so I'll play around with it a bit later here and let you know of any other suggestions.
|
|
|
Re: MLIST - JSON
#22280
09 Feb 17 09:50 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Updated - Im now getting parameters - I did not realise the parameters had to be one long string with commas not individually.
What PPN does AG_XFUNC start in locally?
|
|
|
Re: MLIST - JSON
#22281
09 Feb 17 09:54 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Cross posts - ah Fn'ATE'SBX() sounds useful! - I shell take a look a little later... I think im happy to transfer the file back via ftp once its been created, rather than one large parameters .
|
|
|
Re: MLIST - JSON
#22282
09 Feb 17 10:13 AM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
ATE's default local directory is DSK0:[1,4] on the PC side. So that would be where any file created would end up, absent an explicit PPN. But note that the standard XCALL search path applies just as it would in the server environment, so the SBX should be sent to the %ate%\dsk0\007006 directory.
And, if you are creating temporary output files (like the CSV output of your routine), you may want to put them in the %ATECACHE% directory. Note that %ATECACHE% is a client-side environment variable, but you can use it from the server side in the AG_FTP command to retrieve the file back again.
|
|
|
Re: MLIST - JSON
#22283
09 Feb 17 12:11 PM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Thanks all useful info, I'll plough on tomorrow with it. Was a great suggestion of yours using a local SBX to get around the AIX limitation
|
|
|
Re: MLIST - JSON
#22284
10 Feb 17 04:04 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
BTW, I noticed line 99 of fnatesbx.bsi says xcall MIAME, MX_GETVER and errors, not xcall MIAMEX - Unless i missed the obvious.
|
|
|
Re: MLIST - JSON
#22285
10 Feb 17 06:32 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Thanks for the help, Its all working in a variation of different ways. It will now check the server version to decide if it can do it otherwise it will transfer the json file to the local PC's %ATECACHE% execute the local SBX and then transfer the results back.
(Guess I should now test it on a AIX server!)
|
|
|
Re: MLIST - JSON
#22286
10 Feb 17 06:35 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Final source (so far) and example can be found HERE.
|
|
|
Re: MLIST - JSON
#22287
10 Feb 17 11:24 AM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
Nice work, and sorry about the XCALL MIAMEX typo. (Must have been an editing error after testing since obviously it couldn't have run that way.) I've just reposted an update/corrected version here: fnatesbx.bsi
|
|
|
Re: MLIST - JSON
#22288
10 Feb 17 11:55 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
Thanks again, have a great weekend.
|
|
|
Re: MLIST - JSON
#22289
22 Feb 17 12:24 PM
|
Joined: Jun 2001
Posts: 11,794
Jack McGregor
OP
Member
|
OP
Member
Joined: Jun 2001
Posts: 11,794 |
FYI, regarding the Fn'ATE'SBX() routine, I posted an update to the fnatesbx.bsi module that now supports a Fn'ATE'SBX$() variation for remote SBX functions returning a string argument. And here's an example of using it (to retrieve the MX_OSVER info relative to the ATE client) : osvercli.bp
|
|
|
Re: MLIST - JSON
#22290
23 Feb 17 01:47 AM
|
Joined: Sep 2003
Posts: 4,158
Steve - Caliq
Member
|
Member
Joined: Sep 2003
Posts: 4,158 |
|
|
|
|
|