I am not sure it is worth spending much time on, but I wanted to post here about the decreased performance we are seeing with the newer builds of Linux A-Shell. Below are comparisons between A-Shell 6.5.1721.3 EL7 and 7.0.1760.0 EL7 using a test program which I have attached (bas extension changed to txt) for reference.
Code
.ver
-- A-Shell Version 6.5.1721.3 Up and Running --
.run test3
--------------- TEST 1 ---------------
0.000028 secs Open file
4.292558 secs Iterate file
0.000019 secs Close file
--------------- TEST 2 ---------------
0.000015 secs Open file
4.284644 secs Iterate file
0.000019 secs Close file
--------------- TEST 3 ---------------
0.000015 secs Open file
4.278623 secs Iterate file
0.000019 secs Close file
.ver
-- A-Shell 7.0.1760.0 Up and Running --
.run test3
--------------- TEST 1 ---------------
0.000030 secs Open file
6.549426 secs Iterate file
0.000022 secs Close file
--------------- TEST 2 ---------------
0.000017 secs Open file
6.523732 secs Iterate file
0.000019 secs Close file
--------------- TEST 3 ---------------
0.000016 secs Open file
6.533457 secs Iterate file
0.000020 secs Close file
If you did want to look into it, I could send you the random data file that I was using for testing. The compiler seems to have experienced a similar slowdown as well.
That is a pretty significant difference! And I can't think of any good explanation for it, especially since your test code really only consists of reading file records with a few simple IF statements. The compiler would be a different animal entirely. It's not surprising that it gradually gets slower as it gets more complex, but I wouldn't expect anything like the 50% increase in execution time like you're seeing in your test above.
So yes, I do want to look into it. But before you send me your 320Mb file, let me see if I can reproduce this with my own performance tests...
Ok, I can reproduce this here. The problem appears to main affect FOR/NEXT loops, which suddenly got a lot slower in 7.0.1756.9. I'll get to the bottom of it...
Last edited by Jack McGregor; 25 Aug 2404:39 PM. Reason: Correct edit from 1757.9 to 1756.9
This was an interesting learning experience. (And thank you for reporting the issue!) Here's the story: A few months ago (circa 7.0.1756) an intermittently recurring case of an unexplained crash led to several rounds of code analysis and refactoring in an attempt to eliminate possible openings for memory corruption. One of the advice/warnings that I kept encountering was to explicitly initialize all variables at the time they are declared (in order to eliminate non-deterministic scenarios where a rogue pointer strays into an uninitialized variable's memory). So, I started doing that in the course of other code cleanup.
Somewhere along the line of this process, I was no longer able to reproduce the unexplained crash, although I was never able to pin it down to a particular bit of source code (partly because it was so intermittent, and partly because the crash actually occurred inside of a Windows DLL that was processing a message from the XTREE DLL during destruction of the tree, so it wasn't even something that could be isolated to a specific line of source code).
Anyway, while focusing my attention on that issue, I failed to notice that the internal routine that sets up access to an ASB variable (typically called one or several times per ASB statement) had gotten somewhat slower. It wasn't that noticeable in typical programs, but as you discovered, it was quite noticeable in certain kinds of tests focused on iteration. Turns out it wasn't the iteration per se, just that iteration itself involves repeated access to the loop variables, and if the loop body didn't contain much else, then the variable access overhead started to dominate. (Interestingly, at least for us old-timers who remember the days when disk I/O vastly overshadowed all other performance considerations, in your loop that was reading a file, the file overhead was negligible.)
After reviewing the code changes, I eventually pinned it down to the pre-initialization of a pair of 512 byte buffers. I'm not sure exactly how the C compiler accomplishes that (presumably a block operation rather than a for-next loop), but apparently it's a relatively slow operation. Since it wasn't really necessary in this context anyway, I changed it back to just setting the first byte to null and the performance reverted back to the way it was, at least most of the performance tests I ran. If you want to verify that, download the 7.0.1762.0 version (same one referenced in your DIMX post) ...
This is purely a runtime adjustment, so it doesn't require recompilation. And it has no effect on compiler performance, which you suggested has also declined recently. In reviewing that, I believe it to be unrelated. At least in my tests, it seems to have coincided with the 1046 compiler update (7.0.1757.5), and however ironically, is likely to be the result of adding performance statistics to the LSX file. (The statistics are being captured regardless of whether you actually create the LSX file, so there's currently no easy way to avoid it. And unfortunately, while those statistics would be ideal for analyzing this issue, since they didn't exist in prior versions, they're no use in this particular case.) In any case, compiler performance is much less of a concern than runtime performance, but if you think it's worth further investigation and have a particular LSX example that you think compiles significantly slower in the current 1046 compiler compared to an earlier version, send it to me and I'll take another look.
I just noticed that the version resource in 7.0.1762.1 executable posted yesterday was still set to 1761.1. It doesn't affect any code operations, but does cause confusion (showing the wrong version in ashlog, in systat/v, etc.), so I recommend re-downloading it (from the same links shown above).
That is a very interesting explanation of the cause! Thanks for the fix as it is much appreciated. I can confirm that performance with the test3 program is back to around what it was with 6.5.1721.3 (4.39 seconds). As far as the compiler, I did want to mention it along with this since it also had a noticeable difference between the versions. As you stated though, it is not as big of a concern as the runtime (and I would not spend your time on looking into it.) Unrelated to the A-Shell update, compilation has gotten much slower for some programs on our system. One of the largest only used to take around 5 seconds when Carl hired me 8+ years ago. Now, it takes around 12 seconds on 6.5.1721.3. Perhaps the new statistics could help determine why. I suspect it is either because of an ever increasing number of includes.
Thanks for the confirmation that we had some success here. As for the compiler, it wouldn't surprise me that it has gotten slower over the course of years. For one, it has steadily gotten more complex. And certainly an ever-increasing number of includes and functions, many of which have ever-longer names, all makes for more work. Just to take one example (Stephen's PRCCSTDSP), you can see that it's doing a lot of label, function, symbol, variable and structure lookups...
Code
Performance Statistics *:
======================================================
Phase 0 (/p, /px prescan): 1.1250 secs
Shake out unused routines (/px): 0.0625 secs (3615 kept, 7716 dropped)
Phase 1 (main compilation): 2.9531 secs
Phase 2 (emit RUN code): 0.2812 secs
Total elapsed time: 4.9443 secs
Labels/Functions: 99770 searches, 625.0000 ms (40513 total defined)
Symbol Definitions: 549908 searches, 4218.7500 ms (26148 total defined)
Variable Definitions: 261745 searches, 2343.7500 ms (102807 total defined)
Structure Definitions: 15036 searches, 156.2500 ms (5581 total defined)
But it's a pretty large program -- over a hundred thousand variables and forty thousands labels/functions! And most of them are 40+ characters long. (Perhaps we should just be glad we have modern processors!)
"Symbol definitions" refers to the routine that looks up a symbol to see if it was DEFINEd. That's normally the first step in identifying a non-keyword token, after which it will check if it is a variable. (So the symbol search count is almost always more than the variable definition search count.) Each of these different types of symbols is stored in a separate tree structure whose design is somewhat skewed towards memory efficiency (being as the algorithm is rather old). The nodes of the tree relate to individual characters, so as the symbols get longer (and yours tend to be quite long) and the number of them increases, the search time increases (more or less linearly). It probably could be redesigned to be faster and use more memory, especially when the symbol count is so large, but again, it's hard to say in advance how much it can be improved and whether the tradeoff would be worth it.
I've done a little research into the compiler performance (i.e. how fast it compiles ASB code), comparing the following platforms:
CentOS 7 / A-Shell 32 bit
CentOS 7 / A-Shell 64 bit
Debian 12 / A-Shell 64 bit
Windows 11 / A-Shell 32 bit
All the tests were using 7.0.1762.4 on the same hardware (my development laptop). The Linux versions were all VMware virtual machines, configured similarly.
Here are my findings to far:
The 64 bit version under CentOS7 actually compiles about 10% faster than the 32 bit version. (That is unexpected and probably merits more testing.)
Debian 12 is about the same performance as CentOS 7 (64 bit).
The 64 bit version requires about 50% more memory though.
All the Linux versions are about twice as fast as the Windows version.
I was able to squeeze another 10% gain under Linux by adjusting the GCC (C compiler) optimization options. (That'll probably become the standard but I need to do more analysis to see what the tradeoffs are.)
Most of the time is spent defining and looking up labels, symbols and variables.
We can probably improve the performance by replacing the current linked-list / tree structure algorithm for handling the string lookups with some kind of hash table, at the likely expense of requiring more memory. But it's not yet clear how much improvement is possible, or how complicated the migration will be. I plan to do more investigation before making a decision.
