Ordinary comma delimited data is easy to parse with INPUT CSV, but what do you do when the comma delimited file contains commented lines? A good example is the A-Shell INI.CLR file, which looks something like this:
; Color Definitions (for a primarily blue background)
; Color legend: 0=blk, 1=wht, 2=blu, 3=mag, 4=red, 5=yel, 6=grn, 7=cyn
; VUECLR=override?,edit fg,edit bg, command fg, command bg, help fg, help bg, status/info fg,bg
; Use the following for a reasonable scheme using blue background.
VUECLR=Y, 1,2, 5,2, 7,2, 6,2
; Use the following to disable VUE colors, i.e. use current colors only.
;VUECLR=Y,-1,-1,-1,-1,-1,-1,-1,-1
; EZCLR=override?,text fg,bg, border fg,bg, cmd fg,bg, sts fg,bg, help fg,bg, highlight fg,bg, menu fg,bg, brief menu fg,bg
EZCLR=Y, 1,2, 1,2, 5,2, 0,9, 1,3, 1,4, 5,2, 7,2
; INFCLR=override?,display fg,bg, edit fg,bg, negative fg,bg, update fg,bg, message fg,bg, original msg line fg,bg, forms fg,bg
INFCLR=Y, 5,2, 1,10, 4,2, 7,2, 7,2, 1,2, 0,2
; MMOCLR=Y,border fg,bg, text fg,bg, arrows fg,bg, prompt fg,bg, status line fg,bg, protected fg,bg
MMOCLR=Y, 6,2, 5,10, 6,2, 7,2, 1,2, 7,2
; SCNCLR=override?,screen text (fg), screen background
; The following sets up white on blue
SCNCLR=Y,1,2
This seems like a simple enough format, but the unpredictable presence of the commented lines (marked by a semicolon), and the equal sign make it difficult or impossible to use INPUT CSV. STRTOK comes in handy here because it can be used on a string, allowing you to first input the entire line to check for commented out lines, then pass it to STRTOK. Also, it eliminates the need to deal with special tests for prematurely terminated lines, lines with comments in them, uneven spacing, etc. The following code parses the input, putting into the MAP structures:
MAP1 VUECLR'OVR,S,1
MAP1 VUECLR(4,2),S,1 ! 4 pairs
MAP1 EZCLR'OVR,S,1
MAP1 EZCLR(8,2),S,1 ! 8 pairs
MAP1 INFCLR'OVR,S,1
MAP1 INFCLR(7,2),S,1 ! 7 pairs
MAP1 MMOCLR'OVR,S,1
MAP1 MMOCLR(6,2),S,1 ! 6 pairs
MAP1 SCNCLR'OVR,S,1
MAP1 SCNCLR(2),S,1 ! 1 pair
MAP1 PLINE,S,100
MAP1 STATE,F,6,1 ! 1)VUECLR, 2)EZCLR, etc.
MAP1 RDELIM,S,2
MAP1 FDELIM,S,3,"=,"
MAP1 OPCODE,B,1,0
OPEN #1, "LIB:INI.CLR", INPUT
LOOP:
INPUT LINE #1, PLINE
IF EOF(1)=1 AND PLINE="" GOTO PREMATURE'END
xcall TRIM,PLINE,1
IF PLINE="" OR PLINE[1,1]=";" GOTO LOOP
OPCODE = 0
ON STATE CALL PVUE,PEZ,PINF,PMMO,PSCN
IF STATE < 5 GOTO LOOP
PREMATURE'END:
PRINT "PREMATURE END OF FILE"
DONE:
CLOSE #1
END
PVUE: ! parse VUECLR line
RDELIM = ";" + CHR(10)
xcall STRTOK,OPCODE,PLINE,FDELIM,VUECLR'OVR,RDELIM, &
VUECLR(1,1),VUECLR(1,2),VUECLR(2,1),VUECLR(2,2), &
VUECLR(3,1),VUECLR(3,2),VUECLR(4,1),VUECLR(4,2)
STATE = STATE + 1
RETURN
PEZ: ! parse EZCLR line
RDELIM = ";" + CHR(10)
xcall STRTOK,OPCODE,PLINE,FDELIM,EZCLR'OVR,RDELIM, &
EZCLR(1,1),EZCLR(1,2),EZCLR(2,1),EZCLR(2,2), &
EZCLR(3,1),EZCLR(3,2),EZCLR(4,1),EZCLR(4,2), &
EZCLR(5,1),EZCLR(5,2),EZCLR(6,1),EZCLR(6,2), &
EZCLR(7,1),EZCLR(7,2),EZCLR(8,1),EZCLR(8,2)
STATE = STATE + 1
RETURN
PINF: ! parse INFCLR line
RDELIM = ";" + CHR(10)
xcall STRTOK,OPCODE,PLINE,FDELIM,INFCLR'OVR,RDELIM, &
INFCLR(1,1),INFCLR(1,2),INFCLR(2,1),INFCLR(2,2), &
INFCLR(3,1),INFCLR(3,2),INFCLR(4,1),INFCLR(4,2), &
INFCLR(5,1),INFCLR(5,2),INFCLR(6,1),INFCLR(6,2), &
INFCLR(7,1),INFCLR(7,2)
STATE = STATE + 1
RETURN
PMMO: ! parse MMOCLR line
RDELIM = ";" + CHR(10)
xcall STRTOK,OPCODE,PLINE,FDELIM,MMOCLR'OVR,RDELIM, &
MMOCLR(1,1),MMOCLR(1,2),MMOCLR(2,1),MMOCLR(2,2), &
MMOCLR(3,1),MMOCLR(3,2),MMOCLR(4,1),MMOCLR(4,2), &
MMOCLR(5,1),MMOCLR(5,2),MMOCLR(6,1),MMOCLR(6,2), &
MMOCLR(7,1),MMOCLR(7,2)
STATE = STATE + 1
RETURN
PSCN: ! parse SCNCLR line
RDELIM = ";" + CHR(10)
xcall STRTOK,OPCODE,PLINE,FDELIM,SCNCLR'OVR,RDELIM, &
SCNCLR(1),SCNCLR(2)
STATE = STATE + 1
RETURN
The example above was complicated by the fact that every line of the file contained a different format and needed to be put into different variables. So we needed a lot of individual STRTOK commands. But we did not need any complicated substring processing–it was all very straightforward.