 Filehooks and delete event
#36602
07 Sep 23 07:27 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
We are using Filehooks and are seeing apparent corruption of the pre-record when the delete event has taken place. I have checked the first few bytes of the pre'record on entry to our SBX program that is called on any 'hooked' event occurring, and they don't tally with the expected contents. As the Hook event is post ISAM 6 process, I can understand why the second byte might be zero, but the changes are much more than one byte.
This is on Ashell 6.4 with ISAM.1 files.
Is this a known problem (possibly fixed in 6.5), is it a new problem not previously picked up, or perhaps, is something else going on?
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36603
07 Sep 23 08:10 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Actually, I can confirm that we are seeing the same behaviour in 6.5.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36604
07 Sep 23 08:43 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Reviewing the code, the pre'record buffer is being passed to the hook routine by way of a pointer to the last expression result. That should be equivalent to the actual record, but on further consideration, it may be that some lower level adjustment to the expression handler subsequent to this being introduced (in Feb 2015) has broken that dependency. I'll set up a test and do a more thorough review later today and see if there's a fix needed.
Just to make sure we are in sync, this is an SBX-type hook, right? With the HFE_ISAM_DEL event?
On the likely possibility that a patch is required, which platform(s) are you most interested in?
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36605
07 Sep 23 09:19 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Yes, HFE_ISAM_DEL event passed to SBX routine.
Most important would be 6.5 on Debian.
We could need it on CentOS 6.4 too, but I don't think there is any immediate requirement.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36606
07 Sep 23 11:15 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Regarding Debian, I recently updated my Ubuntu platform from 20 to 22 (LTS). I think 22 is based on Debian 12 rather than the prior 11. But I don't have an easy way to test if executables generated under the current version work on the prior version. If you can't find anything to watch on TV and can't stand the suspense, you could run a test to see if the latest VUE standalone worked ... vue-3.2.280-u22-x86_64.tzIf would be ironic if it didn't, since one of the larger motivations for migrating to Debian from CentOS is the presumed greater stability. But backwards compatibility is always tricky.
Last edited by Jack McGregor; 07 Sep 23 10:19 PM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36607
07 Sep 23 10:32 PM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
I added a hash code to the FHOOKTST3 / FHOOK1 test program to verify that the version of the record passed to the hook SBX was identical to the one in the file prior to deletion, but wasn't able to reproduce any discrepancies. (Note that although the deleted record does get modified when it is added back into the free chain, that doesn't happen until after the hook has been called, so the version seen by the hook should be identical to the version of the record prior to the ISAM 6 operation.) I did, however, make an adjustment to the code to remove what could have been an opportunity for a discrepancy, so perhaps that will have some beneficial effect. And in the process of testing, I found and fixed an unrelated issue with call stack tracing. And there were also some recent bug fixes related to the 64 bit migration, so probably not a bad idea to update in any case. I'm not yet sure though that it's worth disturbing the 6.4 version, at least until we get a sense of whether the hook issue still exists. Here are the relevant updates: ash-6.5.1740.2-u22-x86_64-upd.tzfhook1.bp (sample hook sbx, now with a hash of the record) fhooktst3.bp (sample program using the hook) ash65notes.txt
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36609
08 Sep 23 06:43 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
I am getting a compile error when trying to compile FHOOK1.BP
I haven't updated Ashell yet, which could possibly be an issue, although the error doesn't obviously point to that.
Ashell version 6.5.1725.1/64 running on Debian 11.7. Latest vue appears to run fine.
I am logged to dsk0:[908,50] and running the following command:
.compil fhook1.bp -rc -i -w:4000 -x:2 -p -n -lf -b -l
with the following output:
Phase 1 - Parse source and generate object code
Copying from ASHELL.DEF[907,16]
Copying from TYPES.DEF[907,16]
Copying from HOOK.DEF[907,16]
Copying from FNMINASVER.BSI[907,10]
?Exceeded maximum ++INCLUDE nesting (40) (+65) - ++include'once ashinc:ashell.def
Segmentation fault
I'm guessing that the error message is not truly indicative of the real problem.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36610
08 Sep 23 07:16 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Ashell COMPIl version: 6.5(1013)
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36611
08 Sep 23 08:17 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Interesting. Apparently that has something to do with using dashes rather than slashes for the switches. The dashes are necessary with the standalone compil executable, but I'll have to admit that I wasn't aware that they were even supported by COMPIL.LIT. And apparently there is an issue there, because changing the command line to... .compil fhook1.bp /rc /i /w:4000 /x:2 /p /n /lf /b
... works much better. Although there's still an error which requires upgrading the /p to /px (which you should always do anyway; in fact I'm considering making that automatic). Also, I think /l and /lf are the same, although there's no harm in redundancy here. I'll need to investigate the issue which appears to be within COMPIL.LIT's processing of the switches. The standalone compiler may also have an issue which I need to investigate.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36612
08 Sep 23 08:25 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Ah, apologies. I didn't think about the switches. I'm used to using in APN and Linux which, rightly or wrongly, I tend to use dashes all the time. In fact I think I probably use dashes in Ashell, on those ocasions when I use it, but then it would usually be 6.4.
Also, I do now use -(/)px now, but cut it down in case that was causing the problem for some reason.
Last edited by Stuart; 08 Sep 23 08:27 AM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36615
08 Sep 23 08:39 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Well there's no good reason why it shouldn't support the dashes, but clearly there's a disconnect somewhere.
As for /px, I guess that's the reason why we still separate it from /p, i.e. that it does invoke a lot of extra logic to implement the shake-out of unused functions, so it seemed to make sense to have an option to turn it off. It has been a long time though since I run into any issue with it, except ironically here (you seem to have the magic touch for finding just the right rock to look under), I encountered an issue with this very program in which it was shaking out a function (from callstack.bsi) that actually was being called. I still need to figure out what was going on there, but the simple workaround was just to add "Dynamic" to the function definition. (Dynamic functions have to be immune to the shake-out because there's no way for the compiler to know if the function will be called dynamically.) Using /p instead of /px would have been another workaround, except /px seems to be required here for another reason. Yet another workaround in this case would be to just remove the ++include callstack.bsi entirely, since fhook1 relies on a symbol defined within the bsi to decide whether to call it. That little detour did lead to fixing an issue with MX_CALLSTACK.
But all of that is independent of the original issue, so as the French say, revenons à nous moutons!
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36617
08 Sep 23 08:47 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
I now have a missing include: ISPTSTH.DEF.
Can you identify the location for this please?
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36618
08 Sep 23 09:06 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
That should be in 908,50 along with the FHOOK1. You can download it from the repository: ISPTSTH.DEF
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36619
08 Sep 23 09:11 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
That is what I presumed but only ISPTSTH.BP is listed, ISPTSTH.DEF isn't.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36620
08 Sep 23 09:14 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
BTW, the problem with dashes on the COMPIL.LIT command line is that all of the switches are being ignored! (You get the same error if you just try .COMPIL FHOOK.BP)
Since it would be nice to not have to toggle the brain/finger connection each time you switch between environments, I'll adjust that to support either lead-in. But that'll have to be next week.
Back to FHOOKTST3 and FHOOK1, the program only does one ISAM 6, so it's not exactly a torture test. But to save you from further investigative detours, FHOOK1.SBX outputs a log to OPR:FHOOK1.LOG. The entries now include a digest at the end of the line, based on the full pre'record passed. FHOOKTST3 also creates a log, FHOOKTST3.LOG, which also lists the md5 digest of the record before deleting it. It's not exactly user friendly, but typing both logs on the screen should allow you to verify that the digests match. If your issue is intermittent, that might not prove anything. But if so, there is at least some reason to expect that the ashell update might close the window of opportunity for deviation between the two.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36622
08 Sep 23 09:32 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
I tested an alternative of the same. On initial entry to our hook BSX, modelled on FHOOK1.BP, I ouput the asci values of the first 20 bytes of the pre'rec and rec variables. Running that SBX for an addition and a delete, the delete does not give the expected values for the record: 2023-09-08 10:18:00 165 Debug: event hooked:2048
2023-09-08 10:18:00 165 Debug: pre'rec:00000000000000000000
2023-09-08 10:18:00 165 Debug: rec:49786987848980693284
2023-09-08 10:20:28 165 Debug: event hooked:1048576
2023-09-08 10:20:28 165 Debug: pre'rec:12864000000341832518
2023-09-08 10:20:28 165 Debug: rec:03803100197017801760
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36623
08 Sep 23 09:40 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Ok I'm intrigued. But confused. Are all of these values being printed by your hook SBX? In that case, the rec value is not reliable, as with this particular event (HFE_ISAM_DEL), the post-rec variable is meant to be ignored (per the history note buried at the bottom of the MX_FILEHOOK topic)...
2015 February, A-Shell 5.1.1402: Added events HFE_ISAM_ADD and HFE_ISAM_DEL. Note that unlike most of the other file hooks, these don't come in PRE- and POST-flavors. For the ADD hook, the record data supplied to the hook routine (rec parameter) will contain the key rather than the record data. Since ISAM adds are followed by a WRITE statement, you can hook that to get the contents of the record itself. For the DEL hook, the record data of the record being deleted is supplied to the hook routine in the pre'rec parameter. The rec parameter should be ignored. The sample hook program FHOOKTST3.BP and hook subroutine FHOOK1.SBX in EXLIB:[908,50] have been updated to illustrate both hooks.
So is this just a misunderstanding? Also, is this 6.5.1740.2 or or still 6.5.1725 ?
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36624
08 Sep 23 09:48 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Now getting missing fn'hooklog'skip'to and fn'eachrecupd messages.
Could you send me a copy of the compiled FHOOK1 and FHOOKTST3 files, so I can run them and confirm the results we get with the same start point as yourself?
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36625
08 Sep 23 09:51 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Yes, aware of the note regarding rec for deletes. The key elements are the first rec and the second pre'rec, which I believe should be the same.
Yes, at this time this is 6.5.1725.1/64
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36626
08 Sep 23 09:56 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Ok, that makes sense, but it's difficult to be sure what happened in the interim. Here are the compiled versions... fhook1.sbxfhooktst3.run
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36627
08 Sep 23 09:56 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Just for completeness and in case I am missing something obvious, the relevant chunk of the code that I have used is as follows: !... previous code
define MAX_RECLEN = 1024 ! max rec size we care about
MAP1 params
MAP2 status,i,2
MAP2 hookenv,ST_HOOK_ENV
MAP2 rec,x,MAX_RECLEN
MAP2 pre'rec,x,MAX_RECLEN
map1 filespec$,s,40
MAP1 testrec,ST_TESTHOOKREC,@rec
MAP1 pre'testrec,ST_TESTHOOKREC,@pre'rec
map1 recx,s,20
!----------------------------------------------
! Call added to prevent function being excluded when using /px compil argument
common'fno = fn'free'fno(common'fno)
CALL READ'ARGS
call fn'log("%miame%/log/hooks/fhooks", "hooksb: called for event:"+hookenv.event+", "+hookenv.prog+", "+hookenv.fileid+", status:"+status)
CALL CONVERSATION
END
!----------------------------------------------
++INCLUDE MAPS:ABESBX.bav
++INCLUDE MAPS:fnhook.bav
++include maps:fnsub.bav
!----------------------------------------------
! READ ARGUMENTS PASSED IN
!----------------------------------------------
READ'ARGS:
xgetargs status,hookenv,rec,pre'rec
RETURN
!----------------------------------------------
!
!----------------------------------------------
CONVERSATION:
call fn'Logger("event hooked:"+hookenv.event+","+hookenv.flags+","+hookenv.fileid+","+hookenv.recno+","+hookenv.pid+","+hookenv.recsiz+ &
","+hookenv.mode+","+hookenv.prog+","+hookenv.sbx+","+hookenv.user,log'debug)
if hookenv.prog = "ISMBLD" return
! Added for testing
recx = asc(pre'rec[1;1])
for i = 2 to 20
recx += asc(pre'rec[i;1])
next i
call fn'Logger("pre'rec:"+recx,log'debug)
recx = asc(rec[1;1])
for i = 2 to 20
recx += asc(rec[i;1])
next i
call fn'Logger("rec:"+recx,log'debug)
!... program code continues
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36628
08 Sep 23 10:15 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Are you absolutely sure that you're deleting the right record #? (Presumably that error would have been obvious long before this!) The code above seems to be logging the hookenv.recno, but the excerpt you posted earlier did not show it. Otherwise maybe it would be helpful to post the relevant bits of code leading up to the ISAM #ch, 6 call (something along the lines of the example at the bottom of the ISAM statements topic), although it might not be so easy if the code is broken up into separated modules. Or, perhaps it's time to see whether the update fixed the issue. Was the update not compatible with your version of Ubuntu/Debian? It shouldn't be too hard to test the update without interfering with your working environment - just save the updated ashell executable under a different name.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36629
08 Sep 23 10:16 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Nothing expected to happen in the interim. I created a record, checked the output was as expected at that stage, deleted the record and re-checked.
Yes, deleting the same record. I did remove some of the output as I didn't think it was relevant, so it won't match the code above exactly.
OK. Using your programs I do get a match of hashes as you did.
However, if I am reading the code correctly, you calculate and print the hash before you delete the record.
My check is within the SBX as the record data is passed into the hook trapping routine.
Last edited by Stuart; 08 Sep 23 10:19 AM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36630
08 Sep 23 10:25 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
I think you are referring to the hash generated in FHOOKTST3 (which is done before the delete). Then the hook trapping routine (FHOOK1) also generates the hash and tacks it on the end of the (long) log line.
But you're right that maybe I'm testing for a slightly different error scenario than you are. I'm verifying that the version of the record passed to the hook routine (in pre'rec) on the ISAM 6 deletion matches the version of that record in the main program prior to the deletion. In contrast, you're testing whether the post WRITEL version of the record in the hook SBX (from the prior update event) matches the version passed to the hook routine prior to the ISAM 6. In theory they should be the same, assuming the program isn't doing anything in the meantime.
I'll update the test program and hook routine to try to simulate your test...
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36631
08 Sep 23 10:27 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Other testing was going on earlier. The system is clear now, so I'll try the newer version of Ashell.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36632
08 Sep 23 10:32 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
I am getting library errors, so it would appear that a simple of copy ashell is not sufficient in this case. ashell: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ashell)
ashell: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ashell)
Last edited by Stuart; 08 Sep 23 10:34 AM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36633
08 Sep 23 10:55 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Ugh. That seems to suggest that Ubuntu 22 depends on a newer version of the GLIBC library. That may require me reloading an Ubuntu 20 environment (or you updating yours!) But since either of those is too big of a project for Friday night, maybe we can try one more round with updated test modules. (Re-download the FHOOKTEST3 and FHOOK1 routines from the links posted earlier.) I added a digest of both the pre'rec and rec parameters on entrance into the FHOOK1.SBX, and also add a CREATE-then-DELETE sequence to FHOOKTST3 to simulate your test. Here's what my updated OPR:FHOOK1.LOG shows: 08-Sep-23,10:38:26,joaqu,FHOOKTST3,,writel+, 1001 , 6 ,ch# 150 ,loc &h6FF
pre'rec digest=8d7d1020185f9b09cc22e789887be328
rec digest=9ed5632f481b4e09dcae5b80b2b8c1a4
(prior data NA) id=038306, name="Person #038306", balance= 999 , limit= 1999.00
08-Sep-23,10:38:26,joaqu,FHOOKTST3,,isam del, 1001 , 7 ,ch# 150 ,loc &h8B3
pre'rec digest=9ed5632f481b4e09dcae5b80b2b8c1a4
rec digest=68bb3ac9d1cbc38a7875bd8b33b4b503
id was=038306,name was="Person #038306",balance was= 999 ,limit was= 1999 digest=9ed5632f481b4e09dcae5b80b2b8c1a4
So the rec digest on the WRITEL event (when the record was created) does match the pre-rec on the subsequent ISAM DEL event. If you don't get that result, it would strongly suggest that the ashell update is needed. Otherwise, it would still be unclear whether we're misunderstanding something about the situation. One anomaly I do notice is that the record # shown in the hook SBX's log for the WRITEL event is one less than for the delete event (6 vs 7, the number just before the ch# 150). The data indicates that it is the same record though, so I think that's a bug in the hook wrapper code (another thing I need to investigate)...
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36639
10 Sep 23 08:42 PM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Ok, I re-installed Ubuntu 20 LTS and generated an updated ashell so you can test this without having to deal with upgrading the OS ... ash-6.5.1740.2-u20-x86_64-upd.tzash65notes.txtAs an aside, Ubuntu 20 LTS (aka Focal Fossa) still has until April 2025 until its EOL, so I guess I was a bit premature in upgrading mine to version 22 (aka Jammy Jellyfish). I'm assuming you are actually using Ubuntu and not some other Debian-reliant distribution, and given the lack of absolute clarity about the relationship between Ubuntu and Debian, I've decided to stick with Ubuntu identifiers (-u20 and -u22) rather than Debian identifiers (-deb11). If that doesn't seem right to you, please say so.
Last edited by Jack McGregor; 10 Sep 23 08:45 PM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36640
11 Sep 23 09:32 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Firstly, we are using Debian itself. I can confirm that I get the same results with your code as you do under Ashell 6.5.1725.1/64. So, I think that would suggest that the issue is somewhere in our Hook program rather than Ashell or the underlying Hook processes. We have the Hook event processor (SBX:HOOKSB.SBX based on the original Hook program from yourself I presume, akin to FHOOK1.BP). This creates an ISAM file of hook event records containing pre and post records. A separate program processes these. I think that is incidental, as we appear to be seeing an issue at the initial entry to HOOKSB. The following output comes from the following process. A record is created with a 'standard' program for creating/modifying/deleting records (CTYPE). That record is then modified and then deleted. A separate program (HKSCHK) reads the records from the same file as a check on the record content at various points. CTYPE (record create)
2023-09-11 15:19:45 306 Debug: adding record
2023-09-11 15:19:45 306 Debug: rec digest pre isam:38e89786f18546d83e44542a8ebb8f7b
HOOKSB
2023-09-11 15:19:45 306 Debug: event hooked:2048,197119,1120,10,306,128,0,CTYPE,,ST
2023-09-11 15:19:45 306 Debug: pre digest:0f343b0931126a20f133d67c2b018a3b
2023-09-11 15:19:45 306 Debug: rec digest:d800b8736895db4e077846597704ce0a
CTYPE
2023-09-11 15:19:45 306 Debug: return from rec update:1NEWTYPE , rkey:10
2023-09-11 15:19:45 306 Debug: rec digest pst isam:38e89786f18546d83e44542a8ebb8f7b
HKSCHK
2023-09-11 15:21:05 49 Debug: reading record:1NEWTYPE , rkey:10
2023-09-11 15:21:05 49 Debug: rec digest:38e89786f18546d83e44542a8ebb8f7b
CTYPE (record change)
2023-09-11 15:21:22 306 Debug: existing record read
2023-09-11 15:21:22 306 Debug: rec digest pst read:38e89786f18546d83e44542a8ebb8f7b
2023-09-11 15:21:28 306 Debug: changing record:1NEWTYPE , rkey:10
2023-09-11 15:21:28 306 Debug: rec digest pre isam:2f0dcb5f1a03decb4edeaf6018d86a8d
HOOKSB
2023-09-11 15:21:28 306 Debug: event hooked:2048,131583,1120,10,306,128,0,CTYPE,,ST
2023-09-11 15:21:28 306 Debug: pre digest:d800b8736895db4e077846597704ce0a
2023-09-11 15:21:28 306 Debug: rec digest:86cd2b32056107390e7c515bc7b8483e
CTYPE
2023-09-11 15:21:28 306 Debug: return from rec update:1NEWTYPE , rkey:10
2023-09-11 15:21:28 306 Debug: rec digest pst isam:2f0dcb5f1a03decb4edeaf6018d86a8d
HKSCHK
2023-09-11 15:21:47 49 Debug: reading record:1NEWTYPE , rkey:10
2023-09-11 15:21:47 49 Debug: rec digest:2f0dcb5f1a03decb4edeaf6018d86a8d
CTYPE (record delete)
2023-09-11 15:21:58 306 Debug: existing record read
2023-09-11 15:21:58 306 Debug: rec digest pst read:2f0dcb5f1a03decb4edeaf6018d86a8d
2023-09-11 15:22:01 306 Debug: deleting record:1NEWTYPE , rkey:10
2023-09-11 15:22:01 306 Debug: rec digest pre isam:2f0dcb5f1a03decb4edeaf6018d86a8d
HOOKSB
2023-09-11 15:22:01 306 Debug: event hooked:1048576,131583,1120,11,306,128,0,CTYPE,,ST
2023-09-11 15:22:01 306 Debug: pre digest:227859363460b712a411c0100469a156
2023-09-11 15:22:01 306 Debug: rec digest:4d3139378859f5ae172ac221a130318c
CTYPE
2023-09-11 15:22:01 306 Debug: rec digest pst isam:2f0dcb5f1a03decb4edeaf6018d86a8d
What we see is the digest for the record pre and pst creaet is the same within CYTPE. This is expected as the digest is calculated on the in memory record. The external check on the record is also the same, which would indicate they are looking at the same data (once read into the same record layout in memory). When the record is changed, the initial read shows the same digest, as we would expect. The record is changed in memory and from that point, follows the same process as for create, so the pre and pst digests are the same (although changed to the read). Again, the check shows the same digest. Lastly, we have the delete, same flow same results, although there is no check as the record has been deleted. Also, although the record has been deleted from the file, the same data exists in memory so the pst delete call digest still retains the same value. Where we appear to have the issue is within the HOOKSB.SBX routine. The create and change appear correct. Possibly not too relevant, but the HOOKSB digest does not agree with the CTYPE and HKSCHK digest for what should be the same data. However, the delete does not appear correct. According to the digest, the data read in for the delete is not the same as the data from the last write (record change), even using the different values that HOOKSB calculates. The digest calculation is early in the routine and directly on the 'rec/pre'rec' variables. I believe your tests are on the testrec variables, so could something be occurring with the raw data that is passed in, prior to it being allocated to a record layout?
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36643
11 Sep 23 11:42 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Hmmm...  I don't think it's relevant to the hook issue, but it probably is worth following up separately regarding your position on the Debian platform, i.e. whether it's wise for us assume that the Ubuntu LTS releases are a good proxy for the Debian releases, why you chose to go with Debian rather than Ubuntu, etc.. (The vast majority of A-Shell/Linux users are still on RHEL/CentOS but are likely considering migrating, so it makes sense to try to achieve some common understanding of the landscape before everyone goes needlessly in a different direction. If you don't want to have that discussion on the forum, perhaps you can talk it over with Matt so he can discuss your position at the Conference.) For now we'll assume that Ubuntu 20 / Debian 11 and Ubuntu 22 / Debian 12 are sufficiently compatible pairs, as far as A-Shell is concerned. You say that you get my results with my test program using 6.5.1725.1. That suggests that the issue is more closely related to your hook subroutine(s) than to the A-Shell core. But, as 6.5.1740.2 does close a potential gap between record buffer associated with the READ and the buffer passed to the hook, I think it would be worth trying that version just to verify that it doesn't fix the issue in your hook. (Offhand I'd say there's about a 40% chance that it will.) And I think there is a 99% chance that the ash-6.5.1740.2-u20-x86_64-upd.tz in my previous post will be compatible with your Debian version. (Copy it to a different name, i.e. ashell_17402 to keep it separate from your existing 1725.1 version.) Another thought is that the issue may have something to do with events within the hook sbx (HOOKSB) themselves being hooked. And/or the fact that the hook subroutine is making what are essentially recursive ISAM calls (something that the original ISAM code surely didn't have to think about prior to hooks being introduced.) It's not quite clear to me from your explanation above though whether you could test that theory by simply disabling the part of your HOOKSB that performs the secondary ISAM operations. Separately from that, given the uncertainties about the digest differences, maybe it would be better to just replace the digest with a hex conversion of the raw record (representing every byte as a 2 digit hex string). That would reveal if the digest difference is being caused by a one or two byte difference vs complete different data. (Your original post seemed to suggest that it was the former, but if so, it might be helpful to see exactly which bytes are being changed, and to what.) Finally, it's hard to say whether the digest discrepancy between the HOOKSB and HKSCHK routine is relevant, but it's like a loud motorbike idling just outside a concert hall, making it difficult to concentrate on the music. The log of your change routine is hard to make sense of, with the digests apparently within HOOKSB indicating that it is looking at different record buffers than the CTYPE and HKSCHK digests. On that last subject, you raise a concern of a possible disconnect between the rec and pre'rec variables in the hook subroutine, vs. the overlaid testrec and pre'testrec record variables which are of a different size. The only reason for introducing that complication into the sample hook was to allow a single hook subroutine to work for multiples files. But if you're not depending on that feature, I suggest eliminating the complication and getting rid of the overlays. That might clear up some of the confusion. If we're still not getting anywhere, feel free to email me your hook routines and ideally your test program (although I could probably create a test program for the hook routine without too much difficulty if yours is too intertwined with other stuff.) Use the LSX versions for maximum simplicity / portability.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36645
12 Sep 23 06:03 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
I'll get some more information regarding Debian/Unbuntu and come back to you.
In trying the newer Ashell I am getting an Error 61, which is past the end of my list (51) and I can't find a newer list. Given this is 6.4 compiled code running on 6.5, I'm guessing the error is coming from that area, but obviously don't know for sure.
I have tried re-compiling on 6.5 but am hitting various include errors, as the 6.5 system is not a complete dev environment and is a Debian system (it is being used for testing running code and is what is used for the deployed customer systems). Our usual development system is a CentOS system, but again, I presume Ashell should take care of that on the whole.
On this point, on the basis of your comment above, I presume it should be possible to compile from the LSX file, but have never tried before and am getting similar errors. How would one compile an LSX file?
Last edited by Stuart; 12 Sep 23 07:37 AM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36646
12 Sep 23 06:57 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
I have managed to update a working 6.4 container to 6.5.1730 and the programs have compiled OK. However, this is only in Demo mode as I am getting a message that the version isn't covered under the licence.
Do you have a CentOS update to 6.5.1740 please and I'll see if I can compile in that and then copy to the Debian 6.5.1740 system.
Last edited by Stuart; 12 Sep 23 07:15 AM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36648
12 Sep 23 08:54 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Here's a CentOS update: ash-6.5.1741.0-el7-upd.tzash65notes.txtRegarding the error #61, it's "Invalid reference to a collection". You should be able to get the latest ???msg.* message files by logging into SYS: and using UPDCUR, but if for some reason that doesn't work for you, here's what you're missing: 001,051,First logical unit not mounted
001,052,Illegal key
001,053,Illegal key number
001,054,String overflow
001,055,Undefined array in XCALL
001,060,Illegal assignment to iterator
001,061,Invalid reference to collection
001,063,Func/Proc parameter type/size mismatch
001,064,Unsupported type (S?) or scope (local) of array passed by reference
001,065,External library error
001,066,Mismatched collection types
001,067,Invalid collection operation or reference
001,068,Collection types not supported in this runtime
001,069,ATE comm breakdown, client unresponsive
001,070,Invalid dynstruct reference
001,071,Undefined dynstruct member
001,072,Undefined dynamic function
Ironically that error #61 might possibly have been introduced very recently, and then fixed in 1740.3; to rule that out you may want to update your other 1740.1 copies. (I'll post those updates shortly)... As for compiling an LSX, in theory it's just a matter of explicitly adding the .LSX extension, i.e. .COMPIL FOO.XLS/X:2/M/PX/... There have been numerous updates and fixes to this area though, so it wouldn't surprise me if you're getting errors either using an older version for the compilation, or using an LSX that was created in an older version. If you are able to create an LSX with a relatively current version that doesn't recompile under that same version and are willing to share it, please email it to me and I'll get to the bottom of the issue.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36649
12 Sep 23 09:01 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36650
12 Sep 23 09:04 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Is there a fix for the licence issue? I think I need two numbers. I only have a 3775 number and that doesn't appear to support 6.5.
In terms of compiling LSX files, do the switches apply prior to LSX creation, after or both, dependent on switch?
Last edited by Stuart; 12 Sep 23 09:10 AM.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36651
12 Sep 23 09:40 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Oh, and about the license, you should have a link to a spreadsheet that Ty maintains with all your current licenses. But I've just emailed you a copy of your current in-house license which should resolve that issue.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36652
13 Sep 23 04:08 AM
|
Joined: Oct 2015
Posts: 237
Stuart
OP
Member
|
|
OP
Member
Joined: Oct 2015
Posts: 237 |
Finally I have a solution, although not a full answer. It would appear that the issue was resolved within the various Ashell updates. The Hooks digest still doesn't match the digest for the same record in the program that creates/edits the records. The digest difference may be as simple as how they are stored in memory. In the Hook program the data is stored in 'pre'rec' and 'rec', which are both x variables. Within the create/edit program, the digest was calculated on a string ('s') variable, albeit with a record layout sitting beneath that, starting with an 'x' type: map1 rec,s
map2 recx,x
map3 key,s,8
map3 field2 ...
However, with 6.5.1741, an ascii print of the records does match and the result of the Hook delete event is now as expected. We have the Debian and CentOS updates for 6.5, but we may need the Ashell fixes applied to 6.4 at some point. We will continue for now though, without them, as I don't believe the requirement to look at the hook records for deletions is a common enough to warrant your time, unless/until we are specifically asked for it.
|
|
|
|
Re: Filehooks and delete event
[Re: Stuart]
#36653
13 Sep 23 09:32 AM
|
Joined: Jun 2001
Posts: 11,938
Jack McGregor
Member
|
|
Member
Joined: Jun 2001
Posts: 11,938 |
Well, I guess that's good news. So it does sound like the original issue reported was fixed between 1725.1 and 1741. Based on that, my confidence level in the one-line patch originally discussed being the solution has increased from the previously suggested 40% to 95%. (The remaining 5% of uncertainty covers the possibility of it really having been addressed by some combination of changes from 1725.1 to 1741. That's primarily relevant to the question of how easy it will be to migrate the same patch to the current 6.4.1561. But I'll wait for you to call for that.
As for the digest, I suspect you may be on to it. There are clearly options in that routine regarding the auto-calculation of the size where the data type would matter. But at least in the FHOOK1 routine, the XCALL DIGEST is specifying the size explicitly, in which case the type shouldn't matter. Without being able to see the full XCALL statement you're using in your routine, it's hard to say for sure, but given that we have other evidence proving the data is the same, the difference in the digest must be due to some difference in the options or parameters passed. (Maybe something as simple as the digest type?) I'll leave that up to you as to whether it's worth further investigation. (At some point one has to question the payback on continuing further down the rabbit hole to debug the debugging.)
|
|
|
|
|
