ATE currently has two cache directories:

• Temporary (%ATECACHE%, typically %MIAME%\cache), cleared when the last concurrent ATE session exits
• Permanent (%ATEPERMCACHE%, typically %MIAME\permcache), never cleared (other than manually)

In another thread about displaying downloaded images from a web server, the idea of having an intermediate option, i.e. a cache directory that is cleared after a certain time delay, was raised, which I'd like to revisit here.

Some related points to consider:

1) We already have another cache directory with a flexible retention policy -- the APEX directory (which caches files sent to the APEX previewer via the AUXLOC: pseudo device).
2) The WEBCACHE option also has its own cache directory, %MIAME%\webcache but without a retention policy (manual deletion only).
3) Although this thread is posted under the ATE forum, both of the above cache directory mechanisms work in both ATE and A-Shell Windows. So any enhancement here would ideally reflect that.

But to start with the simpler case of ATE, the simplest enhancement would be to just add a retention policy option to the existing temporary cache directory, i.e. instead of clearing it automatically whenever the last concurrent ATE session on a workstation exits, we could set a minimum retention period (in hours). Offhand, setting it to, say, 96 hours, would seem short enough to not create much extra overhead, and long enough to eliminate some percentage of redundant file transfers to the ATE client. Does anyone see any downside? Or a real need for yet another cache directory? We could make it easy to change the time retention in the ATE Connection Properties dialog, but it's doubtful many users would ever see or appreciate that, so the default value would have the most significant effect. (We could set the default to 0 for no change in current behavior, although I would lean towards something a little longer, like 24 to 96 hours.)

As for the WEBCACHE retention, since currently that has to be set via MX_WEBCACHE or SET.LIT, we could just add another parameter to implement the feature and leave it up to the developer/application.

There is also the issue of whether we need to worry about a maximum cache size, aside from a time limit. If so, what should it's default value be?

Not sure about the default time, but the default max size should probably maintain the current behavior.

A related question about ATSYNC/AG_FTPSYNC, do/could they update the modified date/time on the client machine? The reason I ask is when syncing the files with a directory we want to avoid the data transfer if possible, and updating the modified date/time will keep the cache from timing out.

Good idea, and yes, it is possible to update the various time stamps on file directory entries. But it does raise the question of which of the three time stamps (create, modify, or access) are most relevant to cache retention. When I first starting thinking about this in the context of caching images pulled from a web server, it seemed like the creation timestamp would be the most relevant. Copying or transferring a file should not change the modify timestamp but would normally establish a new creation time stamp (even though that leads to seemingly illogical result of a modification time that precedes the creation time).

Perhaps the last access timestamp is the answer. Linux doesn't support it, but under Windows it does appear to get updated by opening the file (i.e. actually using it), but not by ATSYNC merely confirming that it is current. That could be changed (most efficiently adding a new MIAMEX function to explicitly update the last access time, although we could also just open and close the file). But maybe it's not necessary, or even desirable, to update the time stamp if the file isn't actually being used. I guess it depends on the usage. If they aren't going to get auto-refreshed on demand when required, then it probably is better that ATSYNC makes sure they are present in advance, in which case it really should update the last access date even though it isn't really "accessing" the file. But if the files would get auto-transferred on demand and the objective of ATSYNC was just to get the transfer delays out of the way up front, then maybe it's not worth worrying about syncing them in advance of actually being used. (The ones that are actually being used would keep refreshing their last access timestamps, keeping them from getting flushed.) Maybe it requires another ATSYNC switch. On the other hand, the more complicated we make it, the harder it is to use.

In case you don't know I am working on a project that needs exactly this kind of solution. FYI my program displays pictures in an xtree so the user doesn't need to call up that image on another screen and toggle back and forth. For instance they can be entering the description while viewing the image and a gallery of other images.That requires using ATSYNC.

My thoughts are:
Keep it simple
Max size is may be useful but I am not sure and I agree with Frank "the default max size should probably maintain the current behavior"

The solution I like is " add a retention policy option to the existing temporary cache directory" leaving it all the same so as not to step on anybody else's foot. I don't use ATECACHE anywhere else so it doesn't bother me.

My thought was to ask the user if they wanted to clear older files and then turn on the TimeStamp option in ATSYNC.

I'm tinkering with this now. But like so many other things in our world, it's not easy to keep it simple. For images that are likely to be looked at only once, the time when the image was initially downloaded (which will be the creation time from the client perspective) would be sufficient. But if the images (or whatever the cached files are) get re-accessed on a recurring basis, then maybe would want to base retention on the time they were last accessed, not the time of the original download. MX_FILETIMES will return (or even set) the last access time; but that feature isn't universal across Windows versions and file system types. I think it might even be off by default in W10, although it was on in my W11 machine, and you can turn it on manually (not that it would be practical to do so all your ATE clients.) Assuming it is enabled, in the ATE environment we need the AG_FILESTIMES equivalent of MX_FILETIMES (which I'm in the process of implementing).

Once you have the ability to retrieve (and possibly set) the timestamps for files in the cache, then you can implement your own retention policy, expressly deleting files (perhaps using AG_WINEXEC) not accessed within a certain time period. Perhaps that feature could be added to ATSYNC.

Another option would be to implement a configurable retention policy for ATECACHE (and possibly other cache-related directories). I'm currently exploring the idea of adding an AG_CACHECFG command, allowing the application to set the minimum retention time and perhaps query the total # of files and total size of a particular cache directory. Once set, it could be retained in client's registry and acted on automatically, sparing the application of having to continually monitor it.

Note that even without any new features, you could use AG_WINEXEC to invoke a PowerShell script to delete all the files in your cache directory that haven't been accessed (or written, or created) in a certain number of days, something along the lines of ...

(Remove the -whatif to actually perform the deletion - this just lists the files that would be deleted. You'd probably also want to remove the sleep at the end, but without it, you'll have a hard time seeing if it worked.)

After sleeping on this, I'm concerned that the AG_CACHECFG idea is likely to get too complicated to be attractive to anyone. So I think I'm going back to the original idea of just instituting a standard retention policy of 96 hours (since the most recent update or access to each file), applied to both the ATECACHE and WEBCACHE directories. We can quibble over what the ideal retention time should be, but it's doubtful that having a bunch of single-use files sitting around for an extra 4 days is going to be an inconvenience to anyone, while on the other hand, it will probably eliminate the vast majority of otherwise redundant transfers/downloads for files that are subject to being reused. All without requiring any action on the part of the application/programmer. And if 96 hours doesn't seem like enough, other options remain...

• Switch to ATEPERMCACHE (which never gets automatically cleared)
• MX_FILETIMES can be used to manually update the last access times (keeping the files out of the way of the automatic clearing mechanism)
• We can add a feature to ATSYNC for batch updating of the last-accessed time stamps (assuming you really want that, as opposed to just basing it on the actual last access to each file).

Going the opposite direction, if you want to clear the files faster than the default retention, you can use the PowerShell approach shown above. But it is a bit slow and awkward due to the necessity of launching a PowerShell instance, so I may add a couple of AG_xxx commands to make it easier to manage the cache from the application, i.e. retrieve stats (# files, total bytes), selectively clear based on access/modify/create timestamps, etc. Possibly even add a UI option to the Settings menu -- Clear All Cache?. (It seems doubtful though that users would know what to make of that, so it would likely only get used under the dealer's direction, in which case it would be nearly as easy to just use Windows Explorer to clear it.)

I got ATSYNC, Xtree and AUI_IMAGE working with AteCache. Then I moved on to use Apex so the images stick around a little longer. The AG_GETENV is now returning a blank, On some systems it returned c:\miame\cache.

! *** Get Environment variable to see where Cache is installed *** !
AG_GETENV = 45
! ?TAB(-10,AG_GETENV);"AteCache,1"; chr(127);
?TAB(-10,AG_GETENV);"Apex,1"; chr(127);
INPUT LINE,APEX$
?
?APEX$ : STOP

What am I missing.

First, if you want to get a bit more retention out of the ATECACHE directory, and are willing to be a guinea pig, you can try the beta 6.5.1740.0 ...

ash-6.5.1740.0-w32-upd.zip
ash65notes.txt

You should also use the UPDCUR command in SYS: and BAS: to get the updated LITs and SBXs.

With that, the ATECACHE retention is changed from until-end-of-session to 96 hours, and if your Windows workstation supports the last-access-time feature, the 96 hours will be based on that; otherwise on the last create/modify time.

You can also use the new MX_DIRCLEAN in conjunction with any directory to implement your own local image storage and custom retention. (Note however that if using a custom directory, A-Shell won't know to look there for your images unless you specify the explicit location of files to XTREE, IMAGE, etc.)

Going back to the environment variables, an easier way to test for them is to use the AGCMD command, e.g.

Note: the ,1 is kind of archaic now - it was to deal with the 8.3 mangled version of longer filespecs under older versions of Windows. I don't think you can even get those if you wanted them any more.

There are also separate (but somewhat redundant) SETENV and GETENV commands.

But: ATECACHE is only defined by an ATE session; it will be blank if just running A-Shell locally. There is a similar/parallel cache mechanism/directory called ASWCACHE that you may consider if doing this from regular A-Shell/Windows sessions.

As for APEX, it isn't automatically defined as an environment variable. It's default location is underneath your Documents folder (see Files and Folders under APEX > Other Topics), but you can define it if you want it to be somewhere else. You can use AGCMD for that too...

... although note that setting environment variables using SETENV only affects the current session.

I've lost track of how you are getting the images in the first place, but if you were using the webcache mechanism, then they'd be going into %MIAME%\webcache, which, like ATEPERMCACHE, retains files as long as you want. You can clear it using the new MX_DIRCLEAN, or using SET WEBCACHE CLEAR.