i have some large files (sequential) i am processing and i use mx_filepos to save position, look at next record then pop back to it again. It seems after awhile the position is being returned as -1, which causes me to start at the beginning of the file and basically a infinite loop scenario. any suggestions? thanks

-- A-Shell Version 6.5.1711.5 Up and Running --

input file
.dir bds*.dne
BDSC11 DNE 5289029 DSK8:[65,11]

program output
1213000
1213409 current line
-1 file pos returned
2147482120 last file pos
Program stop at location counter &hA1 of Y.RUN
Enter CR to continue: ^C

program

MAP1 S,S,2048
MAP1 LPOS,F,6
MAP1 FPOS,F,6
? TAB(-1,0)
OPEN #1,"BDSC11.DNE",INPUT
L:
INPUT LINE #1,S
IF EOF(1) THEN GOTO D
XT=XT+1
XCALL MIAMEX,118,1,0,FPOS
INPUT LINE #1 S
XCALL MIAMEX,118,1,1,FPOS
IF FPOS < LPOS THEN
? STR(XT)
? STR(FPOS)
? STR(LPOS)
STOP
ENDIF
LPOS = FPOS
IF INT(XT/1000) = XT/1000 THEN ? TAB(12,1);STR(XT)


GOTO L
D:
CLOSE #1
END
********************************************************************

You've stumbled on a gap in the large file support. MX_FILEPOS is using the stdlib function ftell() which is limited to 32 bit offsets, after which it returns -1.

For a minute I thought we could work around that by using DYNLIB to call another function, probably ftello(), directly. But all such stream functions require the stream pointer argument, and there's no way I can for the application to convert the file channel # to the stream pointer.

So it looks like an update is going to be required. Which platform are you currently on?

looks like

[root@valli-prod1 etc]# cat redhat-release
Red Hat Enterprise Linux Server release 6.6 (Santiago)

EFS

thanks

EL6 hit end-of-life in 2020 and my virtual machine copy seems to have some kind of problem, so I don't think I'm able to actually generate new executables for it any more. Can you check if the -el7 ashell executable will run there? (EL7 expires next year, so maybe it's a good time to think about jumping to 8 or 9?)

point me to it and i'll give it a try

You could just pull any of the -upd.tz versions from the -el7 download directory.
There's even an -efs version tucked in there somewhere.
You don't have to do a full installation,and I recommend un-tar-ing the executable under a different name, just to see if it will launch ok. Otherwise, you can probably use an -el5 version (which for some reason I can still generate updates for.)

so this will be just a test of whether the el7 will work then you will generate a updated version for me? or the el7 version does not have this problem with the large file support?

Yes / No. (We're just testing to see if the -el7 executable is compatible with your CentOS 6; if so, then I'll generate an update for it. Otherwise we may be able to go back to the -el5 version, which for whatever reason I can still update.)

As for the problem in general, it doesn't affect the standard Windows version, which has used 64 bit references throughout. (The 'C' version, on the other hand, is limited to 32 bit throughout.)

Under Linux, file references have long been 64 bit, except in this one case. This was made simpler by a special flag ("LARGE_FILE_SUPPORT") that causes the compiler to convert all the standard file I/O routines to their 64 bit versions. (The code still needs to upgrade the size of the variables used, but that can and was accomplished via a symbolic typedef which evaluated to either 32 or 64 bit integers depending on that flag.) The one wrinkle in this scheme that I somehow overlooked was that, for unknown reasons, it doesn't work for the ftell() function used to retrieve the current file position. Instead we have to use a different function entirely. So that's what I need to do to all the Linux editions, but if you want a quick turnaround, it will be easier to focus on just one edition, as there are some other edits in the pipeline. If there's no rush and you're ok waiting a couple of weeks, I'll get the change into all the editions and you can decide them which one to use.

the el7 version seems to work, so if you could generate ashell executable at your convenience that would be great. i only process these large files twice a year and won't be doing it again till november

Ok, I think 6.5.1730.6 resolves the issue. Beta versions have been posted here:

ash-6.5.1730.6-el7-upd.tz
ash-6.5.1730.6-el7-efs-upd.tz
ash65notes.txt

Note the restriction on EFS-encrypted stream files. (Working around that would be much more difficult.)

thanks since all my files are encrypted guess i'll just watch the files sizes going forward

The workaround for these outlier 2GB+ files would to decrypt them and then disable encryption while doing your special processing that involves MX_FILEPOS. The decryption/encryption shouldn't have the 2GB issue, it's only the MX_FILEPOS (fseek and ftell) that appear to have that limitation. You can disable the encryption just for that one program using XCALL ASFLAG, AF_NOSTREAMEFS. And since the EFS overhead for stream file I/O is rather high, the performance gain from operating in non-encrypted mode will probably more than make up for the cost of file-level decryption and re-encryption.

ah, one of those undocumented features. if you disable efs for stream file, will encrypted isam+ files still be read correctly?

Correct!

i found this one but appears to be a problem with it?


[root@valli-prod1 oedwards]# tar xvzf ash-6.5.1730.6-el7-efs-upd.tz
ashell

gzip: stdin: invalid compressed data--crc error

gzip: stdin: invalid compressed data--length error
tar: Skipping to next header
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Hmmm...

I re-downloaded it and was able to un-tar it without any difficulty...

re-downloaded and now that works, however and there is always a however, no libpcre.so.1 ....

It's not clear to me how your previous version was operating without it. If you run ldd ashell on the previous version, is libpcre.so.1 listed? If not, it must have somehow gotten static linked. But I can't find a static version of the library for -el7. So you could try this:

That should make it show up in your rpm list (if it isn't there already) ...

Yours would presumably show el6 versions, and maybe there's a compatibility conflict there.
Do you actually use any regex functions (e.g. XCALL REGEX, or INSTR with the 4th argument)? If not, then you could try using this version, which has the regex functions commented out, eliminating the dependency on pcre...

ash-6.5.1730.6-el7-efs-nopcre-upd.tz

struck out on all those suggestions, the last one stated

/vm/miame/bin/ashell7: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not fou
nd (required by /vm/miame/bin/ashell7)
/vm/miame/bin/ashell7: /lib/libc.so.6: version `GLIBC_2.17' not found (required
by /vm/miame/bin/ashell7)

time to figure out the update i guess

thanks

Here's the -el5 version if you want to try that. (In general these executables tend to retain their compatibiity better going forwards than backwards.)

ash-6.5.1730.6-el5-efs-upd.tz

Actually, I'm recalling that the above version 1730.6. I'll repost it after tracking down a new problem.

fyi that el5 version did work but did not solve the original problem, mx_filepos still returns -1. thanks

I thought of another solution to the 2GB / EFS MX_FILEPOS problem, and it seems to work now (at least in my testing here) in 6.5.1731.0 ...

ash65notes.txt
ash-6.5.1731.0-el7-efs-upd.tz
ash-6.5.1731.0-el5-efs-upd.tz

and we have a winner, thank you

Fifth try's the charm!