In 6.1 this would return a value if test.txt existed, but not in 6.4 . was there a reason for this change? thanks

MAP1 S,S,64,"TEST.TXT (602)"


LOOKUP S,X
? TAB(-1,0)
? S
? STR(X)
END

It may have seemed like there was a good reason for the change. The issue has to do with how to interpret filespecs as either being AMOS-style filespecs or native filespecs.

The first change occurred in 6.3.1509.0.1 , when it came to our attention that certain perfectly-legal native filespecs were getting truncated due to the false assumption that they were actually AMOS-style, as in the example you cite. But as it turned out, that change didn't break your code, because of an obscure bit of logic that decided that if spaces in the filename were followed by a character deemed to be a terminator (i.e. illegal) within an otherwise legal AMOS filespec, e.g. /<>(){};=, \t\n\r then we would terminate the filespec there, strip the trailing blank(s), and convert from AMOS to native.

Then, a year later, another developer submitted a complaint similar to yours, except where the spaces were prior to the extension. For example, lookup("CASH .DAT") was no longer finding the file CASH.DAT, whereas prior to 6.3.1509.1, and under AMOS, it did. (The idea here is that since spaces are illegal in AMOS filespecs, the name part of the spec be truncated at space, i.e. at "CASH", and then the extension is applied.

We decided to revert to the old behavior for the case where there were trailing spaces between the end of the name part of the spec and the extension, under the theory that even though legal in Windows and UNIX, no one in their right mind would actually want to respect such trailing spaces in a file name. (That was 6.4.1550.1 )

Unfortunately for your case, the 1550.1 fix also tinkered with the logic deciding which characters should be considered terminators, which had the side effect of allowing characters such as "(", which would be illegal in AMOS filespecs, to instead force the interpretation of the filespec as native. So, under that rule, the "(" in your "TEXT.TXT (602)" has the effect of forcing the filespec to be treated as native, where the embedded spaces are perfectly legal. (Unlike the AMOS case where every file has an extension of between zero and three character, in Windows and UNIX, the "." is just another character and thus filename extensions are not inherently part of the filename framework. It is left to the application to decide whether to interpret filespecs as having extensions and if so what semantics to assign to them.)

That's a reasonably defensible position, i.e., that as A-Shell evolves beyond its AMOS origins, it makes less and less sense to limit filespec syntax based on some obscure legacy AMOS rules. (Many A-Shell developers have no real knowledge of AMOS and code around the native directory system and filespecs rather than the AMOS framework.)

On the other hand, because the preservation of backward compatibility is so important (even in rather squirrelly cases like the ones at issue here), I'm inclined to try to split the baby so as to maintain compatibility while not unnecessarily constraining the capabilities going forward. So, I propose restoring the original rule involving the AMOS-illegal characters /<>(){};=, \t\n\r such that we terminate the filespec there, and then removing any trailing blanks, before deciding if the filespec should be converted from AMOS-style to native.

To illustrate the before and after effects, I've created the following files, shown with sizes in bytes, followed by (blocks)...

And here are the results of LOOKUP(fspec$) for the four version variations (with the proposed patch being 1551.7)

The issue you reported is shown in the 2nd line, where up until the 1550.1 patch, "lookup.run (602)" was treated as "lookup.run". Then it was treated as a raw native, and the proposed 1551.7 would put it back (based on the wobbly idea that "(" doesn't belong in a filename?)

The other variations show the differences in handling a space in either the file name or extension, depending on whether followed by a quasi illegal char (e.g. "(") or by a normal char ("cash.ccc xx" and "cash xx.fff").

Note that the proposed patch would not completely revert everything back to the pre-1509 behavior. The two deviations are the space followed by normal character cases ("cash xx.fff" and "cash.ccc xx"). Under the original (and AMOS rules), the former would somehow be translated to "cash.dat" and latter to "cash.ccc", whereas in the context of modern native filespecs, both seem so perfectly acceptable that it seems crazy not to accept them as valid native specs.

Nor do any of the 4 behaviors make perfect sense. Is there any reason to treat "cash(1).fff" as "cash" under quirky AMOS rules, yet treat "cash (1).fff" as native under the current rules? Should we handle the non-AMOS-conforming oddities differently depending on whether they occur in the filename vs the extension? Similarly, given that "(" is not an illegal character in a native filespec, is there any reason to treat it differently than any other, i.e. "cash (1).fff" shouldn't be treated the same way as "cash xx.fff"?

Given how confusing this has become, perhaps we should let it percolate for bit for making any further changes. Maybe we need a switch to emulate these archaic AMOS quirks, and otherwise treat any filespec which doesn't match the AMOS rules as native. But that would definitely be new behavior and thus not a good candidate for 6.4.

One other detail which shouldn't be lost in the confusion: in all cases, you can force the filespec to be treated as native by prefixing it with "./" (works for both Windows and UNIX).

I can see that I've probably obfuscated the problem with too many details. But given that there have been multiple changes over the last couple of years in the way that certain oddball filespecs are handled, it seems important to get some feedback on the best way to resolve the issue.

To boil the issue down to the simplest situation, it comes down to how to handle filespecs which contain a space, and whether it depends on what kind of character follows the space.

I think we all agree that trailing spaces should be eliminated from both the main and extension parts of the filespec, since those are arguably stupid, dangerous (and most likely accidental). And we might possibly agree that spaces in the extension are pretty dubious. But what about spaces in the main part of the filename? Clearly those are perfectly valid and quite common outside of the AMOS context. So, shouldn't that argue for treating any filespec containing an embedded space within the main part of the filename as automatically indicating a native filespec? And would it matter whether the following non-space character was alphabetic or some kind of AMOS-illegal character, like "("?

If it weren't for the problem of backwards compatibility, particular with some really quirky AMOS behavior, I would say that any filespec containing embedded spaces in the main part, regardless of what kind of character follows prior to the extension, should be treated as-is (native). And that we treat a space in the extension as the end of the filespec, after which we can use the existing rules to determine (or guess) if it should be treated as a native or AMOS-style spec. But that wouldn't match any of the versions illustrated above.

Or, as suggested earlier, we can also add an INI option which, when set, reverts the behavior back to the way it was pre-1509.1, and otherwise changes to the new rule just proposed.

Sensing the mounting anxiety of developers wondering how this is going to play out, and with the benefit of some additional soul searching and a long lunch, I've decided on the following multi-part resolution:

1) Revert to the pre-1509 behavior. (This makes the latest 6.4 behavior match that of 6.2, which should provide some peace of mind for those updating from 6.2 to 6.4.)

Note that under that behavior, trailing blanks on the extension are ignored, but any blanks or other AMOS-illegal character, trailing or embedded, in the main part of the file name causes the file spec to be terminated, which means that the original extension, if present, is ignored and the default applies. So "ABC .DEF" and "ABC(1).DEF" are both treated as "ABC", with the default extension depending on the context.

2) Introduce a new miame.ini option, (OPTIONS=FSPECNATIVE) which causes the fspec converter to assume the filespec is native if it contains any of the following characters: <>(){};=/\ (plus space TAB, CR, and NL, and comma unless there is a "[").

In this mode, trailing blanks will be removed, provided not preceded by a directory separator character (in which case it truly is taken "as-is"). Embedded blanks are not removed, even if only followed by a dot and an extension. So "ABC .DEF" will be treated exactly as written, whereas "ABC.DEF " will be treated as "ABC.DEF" (trailing spaces removed), but "./ABC.DEF " will be treated as "./ABC.DEF " (trailing spaces preserved).

Technically this introduces a new feature, which we try to avoid in the stable release, but in this case, we'll consider it part of a fix. The new version will be bumped up to 6.4.1552.0. (The same patch will appear in 6.5.1610.0 also.)

To illustrate the effect of this on the cases previously considered, here is a repeat of the chart given previously, with the 1551.7 column replaced by the two new 1552.0 variations:

I posted beta versions of 6.4.1552.0 (containing this patch) in various flavors if you want to test it...

ash64notes.txt
(beta downloads directory)