Skip to content

About LWP::UserAgent, https and proxy setup

November 2, 2013

Hello

The last few weeks, I’ve been banging my head to use uscan, Pithub and JIRA::Client::Automated behind a corporate firewall. They are all written in Perl and use LWP::UserAgent to fetch information from Internet. At $work, using a proxy is mandatory to connect to Internet. But LWP::UserAgent https connection does not work through a proxy. This bug was reported 10 years ago but is still not fixed.

Here’s my plan to fix (or work-around) this issue, at least for Debian.

Before going further, let’s step back to explain briefly what is https and how proxies work.

https is not a protocol by itself. https is plain http over a socket encrypted with SSL (aka TLS). To create a https connection, the agent must first setup the SSL socket with the server. Then the agent uses http protocol to communicate with the server. All the traffic is encrypted on agent side and decrypted on server side by the SSL layer.

When creating a connection through a proxy, things get a little more complicated. Plain http requests (like GET, POST and so on…) are sent to the proxy as if the proxy was the http server. Then, the proxy forwards the request to the actual server.

From what I’ve read, most proxy servers refuse to plainly forward encrypted data to a web server. First a negotiation to create a tunnel towards the web server must be done by sending a http CONNECT request to the proxy server. The encrypted socket is then set up between the user and the web server through the proxy. Once this encrypted tunnel is set up, the usual http communication can be done.

Let’s go back to LWP::UserAgent. To create a connection over SSL, LWP::UserAgent will use a SSL library to setup the socket. This SSL library can be IO::Socket::SSL or Net::SSL. Direct https connection with LWP::UserAgent works fine with either library.

That said, only IO::Socket::SSL is able to perform correctly the verification of the server name. Net::SSL does not check correctly SSL certificates. For more details, see https://github.com/libwww-perl/libwww-perl/pull/51 and https://bugzilla.redhat.com/show_bug.cgi?id=705044 .

When trying to setup a https connection through a proxy, LWP::UserAgent (<= 6.05) tries to use the proxy like a regular http proxy without going through the CONNECT phase. This does not work.

Thanks to Steffen Ullrich, LWP::UserAgent and LWP::Protocol::https are now fixed in github.

In Debian, libwww-perl 6.05-2 and liblwp-protocol-https-perl 6.04-2 contains the same patch to fix https_proxy and are now uploaded in Debian unstable.

Next step is to provide simple patches to let uscan, Pithub and JIRA::Client::Automated correctly connect through proxies without jumping through hoops.

All the best

[ Edited: I've removed some bad ideas from this blog about using Net::SSL ]

About these ads

From → Debian, Perl

7 Comments
  1. Thanks for the advice. Seemed to help me a lot

  2. I was hoping we could get this finally fixed in *UPSTREAM* and not in some debian package. It’s already too bad too many perl developers spent too much time on debugging this issue, including me. If we get a fix for this in debian, great, but it might get people by suprise when their code which works fine on Debian starts to fail on CentOS…

  3. Tim H permalink

    I think you are looking for LWP::Protocol::connect http://search.cpan.org/~benning/LWP-Protocol-connect/lib/LWP/Protocol/connect.pm to be used in conjunction with LWP::UserAgent.

  4. you might “use Net::SSLGlue::LWP” before “use LWP::UserAgent”.
    This magically fixes LWP :)

Trackbacks & Pingbacks

  1. LWP::UserAgent https proxy now fixed in Debian | Ddumont's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: