About LWP::UserAgent, https and proxy setup
The last few weeks, I’ve been banging my head to use uscan, Pithub and JIRA::Client::Automated behind a corporate firewall. They are all written in Perl and use LWP::UserAgent to fetch information from Internet. At $work, using a proxy is mandatory to connect to Internet. But LWP::UserAgent https connection does not work through a proxy. This bug was reported 10 years ago but is still not fixed.
Here’s my plan to fix (or work-around) this issue, at least for Debian.
Before going further, let’s step back to explain briefly what is https and how proxies work.
https is not a protocol by itself. https is plain http over a socket encrypted with SSL (aka TLS). To create a https connection, the agent must first setup the SSL socket with the server. Then the agent uses http protocol to communicate with the server. All the traffic is encrypted on agent side and decrypted on server side by the SSL layer.
When creating a connection through a proxy, things get a little more complicated. Plain http requests (like GET, POST and so on…) are sent to the proxy as if the proxy was the http server. Then, the proxy forwards the request to the actual server.
From what I’ve read, most proxy servers refuse to plainly forward encrypted data to a web server. First a negotiation to create a tunnel towards the web server must be done by sending a http CONNECT request to the proxy server. The encrypted socket is then set up between the user and the web server through the proxy. Once this encrypted tunnel is set up, the usual http communication can be done.
Let’s go back to LWP::UserAgent. To create a connection over SSL, LWP::UserAgent will use a SSL library to setup the socket. This SSL library can be IO::Socket::SSL or Net::SSL. Direct https connection with LWP::UserAgent works fine with either library.
That said, only IO::Socket::SSL is able to perform correctly the verification of the server name. Net::SSL does not check correctly SSL certificates. For more details, see https://github.com/libwww-perl/libwww-perl/pull/51 and https://bugzilla.redhat.com/show_bug.cgi?id=705044 .
When trying to setup a https connection through a proxy, LWP::UserAgent (<= 6.05) tries to use the proxy like a regular http proxy without going through the CONNECT phase. This does not work.
Thanks to Steffen Ullrich, LWP::UserAgent and LWP::Protocol::https are now fixed in github.
In Debian, libwww-perl 6.05-2 and liblwp-protocol-https-perl 6.04-2 contains the same patch to fix https_proxy and are now uploaded in Debian unstable.
All the best
[ Edited: I've removed some bad ideas from this blog about using Net::SSL ]