Good day -

I would like to FTP a file from a remote site directly to linux. I am unclear if XCALL FTP or FPT2 will accomodate this, vs sending the file to the local ATE/windows client. Otherwise i will use CURL, however i would prefer ashell be in more control of things.

TIA

XCALL FTP2 can be called from the Linux side (or the Windows side) to put/get files to/from an arbitrary remote server. It (like XCALL HTTP) does depend on the libashnet library so you may need to double check that it is present in the path.

The basic GET/PUT functions have been in place since the beginning of 6.4, but there have been updates to the underlying crypto packages. If you're having trouble connecting to the remote system, if it isn't firewall related, then it could be that the remote system is insisting on a later variation of the protocol, in which case you may need to get the latest version of the library.

Also note that while people use FTP as a colloquial verb meaning transfer files, almost certainly you want to be using the SFTP mode (both for security and for firewall simplicity).

Indeed SFTP is the protocol here.

Thanks for the reply. I will gin up an example and give it a go.

Guess this is what you were talking about?

%Unable to load libashnet:
libashnet.so.1: cannot open shared object file: No such file or directory

Right!

I'm guessing this is an old 6.4 system? If so, I think you should be able to use one of these, depending on which Linux version ...

libashnet.so.1.11.160.el5.tz
libashnet.so.1.11.160.el7.tz

Unpack it into the bin directory, then create a symbolic link to it from the standard lib location, e.g.


Double-check that the link works and that all of the related system libraries are available using ldd ...

.ver
-- A-Shell Version 6.5.1690.2 Up and Running --

In that case you'll want the latest one from the 65dev/bin-linux-el5 or -el7 directory ...

libashnet.so.1.12.163.el5.tz
libashnet.so.1.14.174.el7.tz

Getting closer... this is what im getting now. I will add the code snippet below. TIA.

Manual says -12 says "Error During Download'

Request File
Request Complete. Elapsed Seconds: 2
STATUS=[-12], EMSG=[FTP Log: SyncTreeDownload:. DllDate: Feb 1 2018. ChilkatVersion: 9.5.0.72. UnlockPrefix: MCRSABSSH. Architecture: Little Endian; 32-bit. Language: Linux C/C++. VerboseLogging: 0. ]

Code:


PROGRAM'START:
CALL OBTAIN'ERA
END

OBTAIN'ERA:
TRACE.PRINT "Request File"
A=TIME
HOST="sftp.remedicshc.com"
USER="XXXXXXXXXXXX"
PASSWORD="XXXXXXXX"
LOCALPATH="" ! should default to current dir
HOSTFILE="/Outbound/835/*.835"
HOSTDIR="" ! Null will use full spec of HOSTFILE
OPCODE=XFTPF_GET+XFTPF_SFTP
PORT=22
XCALL FTP2, HOST, USER, PASSWORD, LOCALPATH, HOSTFILE, HOSTDIR, &
OPCODE, STATUS, PORT, EMSG

B=TIME
TRACE.PRINT "Request Complete. Elapsed Seconds: "+STR(B-A)
TRACE.PRINT STATUS,EMSG
!
RETURN

That's a tough one to debug, since "error during download" doesn't tell us much. I have a couple of suggestions though:

1. If your EMSG parameter is not at least 1500 bytes long, try adjusting it so that we get more details returned. (Your version above looks truncated.)
2. Try it with an individual file rather than a wildcard.
3. Use a package like FileZilla to get a better visual on what's out there.

If none of those get us anywhere, and you're ok with emailing me the login coordinates, I would be happy to give it try from here.

Thanks Cap - will try your suggestions first.

Good suggestion on EMSG... there was a nice long log. (will add below). Looks like i need to send this to the vendor to provide download permission?

Filezilla shows these files as -rwxrw-r--

FTP Log: SyncTreeDownload:
DllDate: Feb 1 2018
ChilkatVersion: 9.5.0.72
UnlockPrefix: MCRSABSSH
Architecture: Little Endian; 32-bit
Language: Linux C/C++
VerboseLogging: 0
SshVersion: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
SftpVersion: 3
syncTreeDownload:
mode 0: Download all files.
localBaseDir: []
remoteBaseDir: [/Outbound/835/]
syncMode: 0
recursiveDescend: 0
syncDirDownload:
syncOneFileOrDir:
downloadFileByName:
open_failed:
pathToOpen: /XXXXXXXXXXX-05182021.835
errno: 13
osErrorMessage: Permission denied
--open_failed
Failed to open file for read/write (random access)
Failed to create output file on local filesystem.

I'm not 100% sure about that. It kind of looks to me like it's trying to save the file in the local root directory (localBaseDir: [], and pathToOpen: /XXXXX...). Why don't you first try setting the LOCALPATH explicitly to something like /tmp or ./ and see if you get the same problem. Also, this is a good example where an interactive FTP tool like FiileZilla would make it easier to experiment / verify whether the issue really is related to the remote or local server.

Well i am using Filezilla and i am able to download the files manually.

I am trying to download these directly to my local linux folder. (which you approved above).

Let me try to define it in LOCALPATH

All be Dammed! Thanks Cap that did the trick... perhaps it needed to know if it was client or server? Or just didnt know..

Thanks for the expertise.

Sounds like maybe the documentation needs a tweak to say that LOCALPATH doesn't necessarily default to the current A-Shell logged in directory, and should probably be explicit. Did you go with ./ ? Or something more explicit?

Actually on re-reading it, I think the confusion was that in the case of a wildcard get, the localpath needs to be set to the local base directory. I'm going to add this to the Comments of the FTP2 doc...

I just went with "/vm/miame/dsk##/ppn" so i guess explicit.

The addendum sounds good.

Admittedly after all this time i should know this but i have a mental block...

I am able to import these files now so great! Now i am trying to perform an append operation on them and it won't recognize them. Why? They imported as uppercase file name. (I did a wildcard get). What's the best way to get these filenames to lcs or get ashell to find them?

TIA

PS: we are actively working on this project with the other vendor, i could probably request he loads these as lowercase filenames, but before i do i wanted to handle on this end.

I added "./" to the beginning of the filename and append and kill recognized the file

Right - basically you need to give A-Shell some indication that you're specifying a native specification so it doesn't run it through the AMOS-to-native filespec conversion process, which includes folding to lower case. Any slash in the filespec will do it (since those are illegal in AMOS specs). You might be tempted to think that a filespec like "Abcdefg-1234567-XYzQrSTw.exten" clearly must be native, but A-Shell 6.5 allows AMOS-style names up to 72.8, and since AMOS-style names are not case sensitive, using a mixture of upper and lower doesn't mean anything to the converter.

Prefixing with "./" or with an absolute path is the best solution.

Note that if using these specs with a LIT command like COPY, you'll need to quote them. (That's to eliminate the potential confusion over whether the slashes are meant to be switches or not.)

If you want to just convert them to lower case permanently, see Folding Filenames to Lower Case. A-Shell comes with a script trlcs that will fold a wildcard list of filenames to lower case, although note that the sample in the doc is missing the trailing grave accent character at the end of the mv ... command, which should be:


But if you wanted to do this for just one file at a time after transferring it, you could just build the mv command explicitly, e.g. "mv " + fspec$ + " " + lcs(fspec$), and execute it via HOSTEX.

Dumb question - i am doing a wildcard get for the files, which is working fine.

Now i am attempting to delete the files once they have downloaded. When i process the delete command i am getting a status 0 with emsg = "Files Deleted" which sounds good, however they are still on the ftp site. Do i have to perform a delete operation for each file or can I use a wildcard again?

Followup - i am also requesting to delete these files one at a time and it also says its deleted status 0 but they still remain. Is it possible there is no write permission on the ftp site?

Certainly the intent is to allow wildcard deletions, as suggested by this excerpt from the FTP2 doc:



So, write protection certainly sounds like one possibility, although it seems odd that you wouldn't get back an error code. ((Does the EMSG parameter contain anything?)

This is where I would try deleting the file with FileZilla to see if it has the same problem.

On a related aside, I recently dealt with an end user requirement similar to yours, except that instead of erasing the files after downloading them from the remove server, we needed to rename them. (That's actually a better protocol, since it allows the remote server to detect the acknowledgement of receipt, and the A-Shell client to avoid re-downloading them, but also allows for archival on the server in a way that both sides can see.) The rename operation though does not support wildcards, so the procedure was to use XFTPF_DIR to create a listing of the remote files, and then download and rename them one by one.

Good suggestion. I tried in filezilla and it deleted the file... so i guess what can i do as far as a trace to see where it may be failing? Oddly like i said i am getting a confirmation status of 0.

Your thoughts of a rename are good... the setup we came up with is having an /archive folder that the vendor populates with new files every time he populates the /download folder. I simply get the files from the /download folder and delete them after. This more closely mimicks the mailbox concept we use with other ins vendors. I would be happy with the wildcard delete if it works, but for now i am attempting to do it 1 at time.

Here is my delete code in case you can see anything off... thanks.

DELETE'REMOTE'ERA:
TRACE.PRINT "Removing Remote ERA files..."
OPCODE=XFTPF_DEL+XFTPF_SFTP
LOCALPATH=""
OPEN #7001,"REMEDICS.DIR",INPUT
!
PNE'2: INPUT LINE #7001,EFILE
IF EOF(7001)=1 CLOSE #7001 : RETURN
XCALL TRIM,EFILE
IF EFILE="" GOTO PNE'2
!
HOSTFILE="/Outbound/835/"+EFILE : XCALL TRIM,HOSTFILE
TRACE.PRINT "Delete Remote file: "+HOSTFILE
!
XCALL FTP2, HOST, USER, PASSWORD, LOCALPATH, HOSTFILE, HOSTDIR, &
OPCODE, STATUS, PORT, EMSG
!
TRACE.PRINT "Request Complete." : LOGG$="Delete End." : CALL UPDATE'LOG
TRACE.PRINT "Status: "+STATUS : LOGG$="Status: "+STATUS : CALL UPDATE'LOG
TRACE.PRINT "Details: "+EMSG
?#1,EMSG
!
GOTO PNE'2


Results from the FTP2 session: (Status was 0 but it says "files deleted" not 1 files deleted...not sure what to expect here)

< GET Request >
Files download-synced
DCM-EdgeMed-05252021B.835
DCM-EdgeMed-05252021C.835
DCM-EdgeMed-05252021A.835

< DELETE Request >
Delete Start: /Outbound/835/DCM-EdgeMed-05252021A.835
Files deleted

Delete Start: /Outbound/835/DCM-EdgeMed-05252021B.835
Files deleted

Delete Start: /Outbound/835/DCM-EdgeMed-05252021C.835
Files deleted

The mysterious "Files deleted" message is just the title, which in the normal case would be followed by the list of files actually deleted. But since none are being deleted, the list is empty.

The fact that the returned status is 0, and not a negative value indicating an error, suggests that the routine doesn't think an error has occurred. Why? Probably because there is no real difference between the syntax of a deletion using a wildcard and one using a single filespec. In the single filespec case, failure to match anything probably should be considered a special error. But in the wildcard case, failure to match anything isn't really an error.

That leads the question of why it's failing to match. My best guess is that it's because you are specifying an absolute filespec, e.g. "/Outbound...", whereas the Outbound directory may actually be not in the root of the remote system, but relative to the home directory of the user specified in the USER parameter. This is a somewhat tricky detail, since some servers may actually "re-home" the session so that the root directory is really the FTP home and not the true root. But others may not.

I suggest dropping the leading "/" from the HOSTFILE.

No joy on removing the leading "/".

Would you mind if i sent you the credentials to give it a try on your side?

(example sent to your inbox... thanks.)

PS: Sent around 9am pst if you didn't get it for some reason please let me know.. some things can bounce around the nexxus indefefinately

Well it's almost grog:30 here on a long weekend - this is time sensitive it would be great if we could hammer it out on Tuesday. I'll check back to see if you have posted any updates in the meantime. Thanks.

Hey - It's not polite to strut your time zone superiority!

But sorry, I got detained by other crises this morning and am just getting back to this now. Hopefully should be able to figure it out today before the factory whistle blows...

LOL - thanks Cap

Frank’s lagging … we 2 Grogs up already here!

Hey hey, isn't it past your bedtime over there???

Maybe one more grog first

Ok, I'm not yet sure why this is, but I was able to delete the files by splitting up the HOSTDIR and HOSTFILE specs ...


(This may require further documentation updates!)

There are still copies in the Archive subdirectory, so you can repeat the test yourself by dragging them back to the /Outbound/835 directory. (Note however, that dragging them in FileZilla actually moves them, so you want to make sure you copies somewhere else on your machine so you can repopulate the the remote directory for repeated testing.)

Very good! Thanks! Have a great long weekend!