Afternoon gents,

First-off, thanks for the forum account! I am the most recent addition to Tom's Omniledger Dev team, far prettier than he is, but far less competent of a coder. It balances things out a bit.

I come to you with a problem we are having;

We are sending HTTP requests that require a long URL to an external API and the URL seem to be getting chopped when they reach a certain length. I�ve had a look at the A-shell reference documentation and it states the maximum URL length is 1024. Which I assumed was in bytes. I realise that this is probably just referencing the response body itself, not the URL length.

Chunk of debug.le;

Content-Type: application/json; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
Date: Fri, 12 Apr 2019 13:43:46 GMT

--readResponseHeader
statusCode: 404
statusText: Not Found
readResponseBody:
bDiscard: 0
contentLength: 340
numBytesToReceive: 340
--readResponseBody
responseBodySize: 340
--fullHttpRequest

We have confirmed that it isn�t a problem with escaped characters or errant white space. We've had a look at the settings of the server and that also doesn't seem to be the problem. I think this may be in the same vein as HTTP GET Custom Headers.

Is there configuration I am missing or is the A-shell HTTP post intended for shorter URLs?

Thanks in advance for any help.

Welcome Matt! (And if you're anything like you suggest, we'll be looking forward to you single-handedly improving the demographics, on multiple fronts, of our next Conference!)

Also, bravo for starting out here with what sounds like a real problem. For whatever reason, it seems that POST-type requests, where the URL only needs to identify the resource, have dominated the HTTP requests up until your latest project, and I'm being a little slow to get my thinking completely around the kind of massive GET URLs that your service apparently requires.

I'll review the code and get back to you shortly, but I think we are looking at another update to remove that limitation.
I think

Thanks Jack!

We are having to interact with an external service that demands these long URLs. We would prefer to send the information via the request body but for some reason the developer on the other end hasn't built his system that way.

Developers... :rolleyes:

Ok here's the deal: the reason why you often run into fixed length limits like this when dealing with XCALLs in the A-Shell environment is that mapped strings don't require trailing nulls in A-ShellBASIC, but within most external libraries (essentially anything written in C or respecting C data type conventions), a trailing null is essential.

The easiest way to simultaneously ensure the existence of a trailing null on a parameter passed to an external library while also protecting it from unanticipated changes by the library (e.g. parsing may introduce additional nulls or character conversions), is to just copy the parameter you pass to the XCALL into a temporary fixed length local stack buffer.

The two competing alternatives would be to allocate a dynamic buffer or to just pass a pointer to your parameter to the library. The latter is the simplest to implement and most efficient, but means that the application must make sure to allow for a trailing null, and be prepared to reload the parameter if using it a second time (in case it gets modified by the library). Both of those conditions are probably acceptable in your case. (The only way you wouldn't get a trailing null is if you MAPped the URL variable at exactly the number of bytes needed. Side note: dynamic strings, i.e. those mapped as S,0 always get a trailing null.) And it seems doubtful you would try to treat the URL as a constant and pass it again without rebuilding it.

But with POST requests, or with simple GET requests, it's conceivable that an application might treat the URL as a constant and reuse it.

So probably the best solution here is to dynamically allocate a new buffer so as be sure of a trailing null and to protect the original URL from changes.

So that's the plan, but it will require an A-Shell update. Are you ok with going to 6.5 for this project? It's a minor-enough patch that I would be willing to retrofit it into the "stable" 6.4 if necessary, but that's still going to require an update (albeit a minor one), so I'd prefer to leave 6.4 alone if it's all the same to you.

As an aside, while we're on the subject of memory management and since you're new to A-Shell, you might be wondering why we don't just always allocate dynamic temporary buffers for everything all the time (like you would expect in, say, Python). The reason is a combination of two factors that are somewhat unique to A-Shell:

a) The nature of the MAP structures has historically created a mindset where the physical relationship of one variable to another might be more meaningful, i.e. the programmer might be consciously counting on that fact in order to pass entire structures from one place to another as a unit. This is certainly the case with file i/o, where you are almost certainly writing a record at a time rather than a list of fields (i.e. like you would in an SQL environment).

b) It is quite common for a single instance of A-Shell to persist for hours, or even days. (In a typical office environment, a user may log in once and use that same instance all day long, running hundreds of different programs.) In contrast, most executables are typically run for a single operation (the length of a single command, or to process a document, etc.), after which the process terminates and all the memory can be completely relinquished to the OS. A process that lasts a long time, and which has many varied memory requirements, needs to be more careful about the way it manages that memory, lest it end up interfering with the efficient use of the overall system memory. (Traditionally, OS-level memory cleanup routines don't try to clean up memory used and then discarded by a still-running application, so it behooves long-running applications to keep track of the memory they've used and released, so as to re-use previously released blocks when possible. Which just adds another level of complexity and overhead to dynamic allocations, which helps explain why we often try to take the shortcut of using a fixed-length temporary stack buffer or a pointer to the original application mapped variables instead.)

I've gone ahead and posted a couple of beta updates in case you want to verify that they actually solve the problem...

ash65notes.txt
ash-6.5.1660.1-w32-upd.zip
ash-6.5.1660.1-w32c-upd.zip
ashnet-1.12.165-w32.zip
ash-6.5.1660.1-el5-upd.tz
libashnet.so.1.12.165.el5.tz

Note that the A-Shell and ashnet library updates are actually independent. To overcome the 1024 URL length limit you need the A-Shell update but not necessarily the library. The library update resolves an issue that Tom had raised in the thread HTTP GET Custom Headers regarding the response returned with the GET operation on authentication failure. (And for that you don't actually need the A-Shell update.)

That was quick, thanks Jack! I'll have a look at it ASAP.

As a response to the 6.4 / 6.5 argument, 6.5 is fine for us. The new system will be a slow and steady roll out so updating to a new version of A-Shell shouldn't be a problem for our customers (or the support staff).

Cheers!