Screen Scrape Experienced? - fresh eyes needed
Posted: Sat Apr 17, 2010 12:41 pm
I've got a custom bit of software that I've been writing for myself to deal with the complete lack of API that almost all UK couriers and parcel delivery services use but I've hit something of a dead end.
My code scrapes a particular delivery service form which has dynamically assigned field names but constant id values then uses the correct field names along with a curl lib to submit the data I need to provide to get a quote... (eg: weight, country, postcode, etc..) but for some reason the CURL library doesn't appear to be submitting the correct information or is including some kind of tell tale because instead of submitting the form properly it just reverts back to the initial form all over again as if no values have been submitted.
Now, I'm pretty sure a lot of work went into making the form/page I'm trying to work with, very difficult to do this sort of thing with so if anyone has a good grounding in screen scraping and CURL in particular I would be very grateful for any tips and a fresh eye...
I hasten to add this code is intended for my own personal use so, if you don't have the skills, please don't ask for a copy...
My code scrapes a particular delivery service form which has dynamically assigned field names but constant id values then uses the correct field names along with a curl lib to submit the data I need to provide to get a quote... (eg: weight, country, postcode, etc..) but for some reason the CURL library doesn't appear to be submitting the correct information or is including some kind of tell tale because instead of submitting the form properly it just reverts back to the initial form all over again as if no values have been submitted.
Now, I'm pretty sure a lot of work went into making the form/page I'm trying to work with, very difficult to do this sort of thing with so if anyone has a good grounding in screen scraping and CURL in particular I would be very grateful for any tips and a fresh eye...
I hasten to add this code is intended for my own personal use so, if you don't have the skills, please don't ask for a copy...