Topic: Statically linked cURL and SSL for win32

[hans]

I'm in need of help finding software. I'm trying to fend off FTP usage in my company and need a solution to post a file to a webservice (simple REST).
The problem is it basically has to be a drop in no installation necessary solution due to the customers "abilities." If they need to install anything or can't copy and paste the command we give them, I fear they're going to fall back to what we've (in the not me sense) got in the contract, which is FTP (even though I keep saying don't offer FTP, and am a little more than agitated).

My boss found a version of wget that would work except AFIK you can't easily do a file upload (with a parameter name) with it. So I think we're back to the original thought of using cURL since I already have the command available.

The customer is on a Windows server box so it's gotta be win32.

[webwhy]

so you're saying that using curl is not an option since it would require installation in the customer's server?

i don't know how the business end of your REST service is implemented, but if it's just a standard mutipart/form-data post to url (e.g https://tgm.com/customers/100/files), as it should be imho, there are several flash and/or java based web components that handle multiple file uploads very well that can be embedded in a web page.

e.g. a GET request to 'https://tgm.com/customers/100/files;new" would display the web form with the embedded file uploader component, which would POST to the 'https://tgm.com/cusotmers/100/files' url to create a new file.

i've also built a simple web start application addressing this same concern for a client.  Basically allowed drag and drop of single files and showed progress bars for each upload in progress...

[hans]

Yes, curl is likely out since it would require installing curl and openssl (to do HTTPS). The person we're dealing with has never heard of curl before. So if we can get them to replace their call to an FTP executable, it has to be very simple. Thus the preferred standalone ability. cURL by itself would work but it's the OpenSSL that is problematic. We haven't been successful in getting a "packaged" version of cURL and openssl working, but I haven't been able to try it on a fresh box yet (I'm currently installing a trial version of Windows server to try a couple things out).


Our REST service uses the a parameter name to pull the file contents. e.g. customerCsv along with other form parameters (there are multiple form elements).

In curl it's something like this:

Code: [Select]

curl -F messagesCsv=@test-data.csv https://domain.com/appname/import/messages

I might be able to write another method that simply takes a PUT which I think wget would work with, but then I'd lose the ability to allow for other parameters, which might work in the this solution since we can probably get rid of the other parameters on the form. But I'd prefer not to if I didn't have to.

Since this is a scripted process, web forms will do me no good here.


This is putting me in a resistance stance with the higher ups since I argued with them this morning about not bending over for customers and instead being a company that is in front of technology instead of behind. But as often, sales dictate what gets done eventually; and that's hard to argue against even though I try. The big problem is we don't really have an inbound FTP solution but that's what got sold despite my pleading because it was an easier sell. But I digress, we suck it up and deal with what we're dealt right?

[webwhy]

Since this is a scripted process, web forms will do me no good here.

ahh...understood...

i'm assuming you're using the bundled package available for windows from the curl website?  how exactly is it failing? 

We've had issues with php, libcurl, and openssl on windows boxes due how openssl handles thread safety.  The application was seg faulting under high load.  There could be workarounds or it even could be fixed by now in the curl bundles, but it was easier for us to move the application to linux.

you might want to look at "powershell";  it's installed on Server 2008, but available for 2003.  all of my windows friends swear by it.  it's a powerful scripting interface for windows which provides access to most of the .NET class library.  If you couldn't write a simple batch file or provide sample code to him that would accomplish the automated task for your client using powershell, i'd be surprised.

[hans]

Well, there's hope.

I just did a clean install of Windows 2003 server (probably similar to what they'll have since they're running SQL). And simply took the latest version of cURL and a windows version of openssl binaries (0.9.8h) and unzipped them in a \usr\local directory I made (no spaces for windows happiness).

After adding each app in a PATH variable, I tried cURL and everything worked just fine. So I removed the OpenSSL path variables and added the DLL's to the cURL base directory and tried again; still worked. So finally I removed the cURL path variable and tried just running it out of the directory and it still worked as well. So I'm not sure what my boss was doing when he tried it, or if he just didn't have the right libs or if it got confused with an existing install on his laptop but it works for me, so we're going to back to the original plan and have them try cURL. I'm debating on if I should have them download the binary openssl and set that on the path in addition to cURL or if we should just give them a "packaged" version of cURL. I'm leaning towards having them add openssl to their path since I think that's better.

tgm needs more coffee though, and possibly to calm down until the customer has problems with the "installation" of a zip file. Here's crossing his fingers that they don't already have some old SSL libs in the system directory.

Remind me again why we try and use shared libraries these days? I think all apps should be self contained.

[Steve]

/sidebar

whats wrong with ftp?

[webwhy]

whats wrong with ftp?

it's insecure by default;  also, one more hole(s) to punch and/or maintain through corporate firewalls...

also, it sounds like tgm probably has web software that processes the files as they're uploaded behind the web url...he would probably have add support for the FTP protocol to his application...or write a bridge process that would link the two as the files come in via a standard ftp server

[hans]

There are many problems in using FTP as a data integration strategy (and very few advantages IMO). In addition to the already mentioned issues, simple problems like how do you know the FTP upload is done and such are often ignored. Plus, typically, FTP is uploaded once a day discouraging "on-demand" processing. It's an old way to do things and often is only done because so called IT workers don't know any better (an should be whipped IMO). There is no feedback usually either, so if their data is bad, they won't know about it without us creating a file, or emailing them or some other more "complicated" solutions.

I just don't follow the "everybody does it, so we should accommodate" mentality.

And don't even get me started on using CSV's.


My boss's wife's company is just now getting around to implementing SOAP in their company and I shook my head, saying (and he agreed) they're already 5 years behind the current RESTful trend. Not that SOAP is terrible per se but it'll be years before they get around to it; the pitfalls of working for a slow big company. Even now, I have a hard time sticking with XML as a the data format instead of JSON (so we do both) since I think JSON is the new XML.

[Steve]

Well that makes sense. What is soap? I'm learning stuff lol

[hans]

SOAP is a heavyweight XML based data transfer protocol that sits atop HTTP (usually). There are multiple variations of the implementation, which, like current RESTful implementations, makes things difficult to deal with sometimes. Since SOAP had extra "junk" (WSDL's and such) many people favored POX (plain old XML) over HTTP instead, which was similar but easier to implement.
While SOAP and POX can both be RESTful, the current trend is towards lightweight REST interfaces that use basic HTTP methods to interact with the services.
True REST uses POST, DELETE, PUT and GET methods for the actions but since browsers (and AJAX clients) don't really do DELETE and PUT right now, you end up with hybrid approaches, like overloaded POST or a hidden _method parameter.

RESTful services tend to have a certain simple elegance and are easily tested with simple programs like cURL and wget since they are generally designed to be lightweight. This makes them favorable for developers too since you don't have a bunch of libraries to integrate with systems, just basic HTTP abilities.

[VBprogrammer]

What about scp tgm? Not a nice web-service solution but much better than ftp and using psftp would be a drop-in replacement for clients.

Please get started on CSV, personally I find CSV to be a great file format. Easy to parse and use, even with command line tools. Users can create their own 'data views', if they create something which is really useful then they can have it added to an application. I read a blog post which explained it better than I could, I'll see if I can find it.

[hans]

Please get started on CSV, personally I find CSV to be a great file format. Easy to parse and use, even with command line tools. Users can create their own 'data views', if they create something which is really useful then they can have it added to an application. I read a blog post which explained it better than I could, I'll see if I can find it.

Try to model anything with hierarchy or custom fields or variable associations. For example, exporting your contact information from Gmail that allows for various phone numbers.

CSV is OK for table data but as soon as you try and do anything "fancy" it breaks down horribly. And customers (ours at least) always want to do "fancy" things but give us a CSV file that ends up with lots of null fields for columns that have nothing to do with the row. Which might be solved by multiple CSV uploads, but good luck with that one in a big company.

[hans]

To be continued here:
http://www.entropysink.com/forums/index.php?topic=8234.0