Portsnap and Squid

At BSDCan this year I listed to Kris Kennaway describe the FreeBSD package cluster (.pdf). He said he uses a caching Web proxy to optimize retrieval of source code when building packages. This makes an incredible amount of sense. Why download the same archive repeatedly from a remote site when you can download it once, and transparently let other clients retrieve the archive from the Web cache?

I decided I needed to emulate this sort of environment for several of my FreeBSD systems. I use Colin Percival's excellent portsnap to keep my FreeBSD ports tree up-to-date. If one of my systems retrieves the necessary updates through a Web cache, the other systems can get the same files from the Web cache. That saves Colin bandwidth and me time.

I set up Squid using the www/squid port. The only changes I made to the /usr/local/etc/squid/squid.conf file are listed below.

http_port 192.168.3.7:3128
icp_port 0
acl our_networks src 10.1.0.0/16 192.168.3.0/24
http_access allow our_networks

The first line tells Squid to listen to the internal private interface of a dual-home VPN concentrator / firewall / gateway (CFG). I didn't want the external interface of the CFG offering a Web proxy to the world. I set the standard Squid port, 3128 TCP. The second line shuts down the ICP service on port 3130 UDP, since I don't use it. Line three sets up an ACL and defines the networks I allow to talk to the Squid proxy. I tell Squid to allow the IP addresses of remote systems connecting via IPSec tunnel (addressed in 10.1.0.0/16 space), and systems on the internal network provided by the VPN CFG (addressed in 192.168.3.0/24 space). The last line enables the ACL.

Before Squid can operate, I tell it to build its cache directories by running 'squid -z' as root.

Finally I edit my /etc/rc.conf file with this entry:

squid_enable="YES"

With this line added, I can use the /usr/local/etc/rc.d/squid.sh shell script to 'start' or 'stop' or 'restart' Squid. Prior to this I did not know that I needed to modify /etc/rc.conf before the squid.sh script would work.

Once Squid was running and listening on port 3128 TCP, I had a system (192.168.3.12) use the proxy through portsnap. First I set the environment variable which tells fetch to use a proxy.

setenv http_proxy 192.168.3.7:3128

When I ran 'portsnap fetch', I could see the program working through Squid to get the files it needed. Here is an excerpt from the /usr/local/squid/logs/access.log file.

1119988542.131    142 192.168.3.12 TCP_MISS/200 668 GET http://portsnap.daemonology.net/t/
001b134d2c8210e43d2cad5072c8c78a6d21a576c7e14f46e751ba7b5d2474c7
 - DIRECT/72.21.59.250 text/plain


This is a TCP_MISS because this request is new to Squid.

Later I ran portsnap on two other systems, 192.168.3.11 and 10.1.2.1, and saw different Squid results.

1119988702.146     95 192.168.3.11 TCP_MEM_HIT/200 677 GET http://portsnap.daemonology.net/t/
 001b134d2c8210e43d2cad5072c8c78a6d21a576c7e14f46e751ba7b5d2474c7
 - NONE/- text/plain

1119989557.080     99 10.1.2.1 TCP_MEM_HIT/200 678 GET http://portsnap.daemonology.net/t/
 001b134d2c8210e43d2cad5072c8c78a6d21a576c7e14f46e751ba7b5d2474c7
 - NONE/- text/plain

In both cases, Squid provided the request files from its cache, and didn't need to request the files from daemonology.net.

I plan to use this system to let remote sensors perform all of their updates through a central location, my VPN CFG system. That one box will run the Squid proxy and retrieve all necessary files once.

Comments

Anonymous said…
Excellent post, Richard! I run a FreeBSD shop using portsnap. Using squid to cache all the updates is perfect.

I've never set up squid before, so the detailed instructions will be a real timesaver for me. Thanks!
5:33 PM
Anonymous said…
Exactly what I was looking for. Thanks Richard!
3:34 AM
website design New York City said…
This comment has been removed by a blog administrator.
9:56 AM
Post a Comment

Popular posts from this blog

Zeek in Action Videos

Image
This is a quick note to point blog readers to my Zeek in Action YouTube video series for the Zeek network security monitoring project .  Each video addresses a topic that I think might be of interest to people trying to understand their network using Zeek and adjacent tools and approaches, like Suricata, Wireshark, and so on.  I am especially pleased with Video 6 on monitoring wireless networks . It took me several weeks to research material for this video. I had to buy new hardware and experiment with a Linux distro that I had not used before -- Parrot .  Please like and subscribe, and let me know if there is a topic you think might make a good video.
Read more

MITRE ATT&CK Tactics Are Not Tactics

Image
Just what are "tactics"? Introduction MITRE ATT&CK  is a great resource, but something about it has bothered me since I first heard about it several years ago. It's a minor point, but I wanted to document it in case it confuses anyone else. The MITRE ATT&CK Design and Philosophy document from March 2020 says the following: At a high-level, ATT&CK is a behavioral model that consists of the following core components: • Tactics, denoting short-term, tactical adversary goals during an attack; • Techniques, describing the means by which adversaries achieve tactical goals; • Sub-techniques, describing more specific means by which adversaries achieve tactical goals at a lower level than techniques; and • Documented adversary usage of techniques, their procedures, and other metadata. My concern is with MITRE's definition of "tactics" as "short-term, tactical adversary goals during an attack," which is oddly recursive. The key word in the tacti...
Read more

New Book! The Best of TaoSecurity Blog, Volume 4

Image
  I've completed the TaoSecurity Blog book series . The new book is  The Best of TaoSecurity Blog, Volume 4: Beyond the Blog with Articles, Testimony, and Scholarship .  It's available now for Kindle , and I'm working on the print edition.  I'm running a 50% off promo on Volumes 1-3 on Kindle through midnight 20 April. Take advantage before the prices go back up. I described the new title thus: Go beyond TaoSecurity Blog with this new volume from author Richard Bejtlich. In the first three volumes of the series, Mr. Bejtlich selected and republished the very best entries from 18 years of writing and over 18 million blog views, along with commentaries and additional material.  In this title, Mr. Bejtlich collects material that has not been published elsewhere, including articles that are no longer available or are stored in assorted digital or physical archives. Volume 4 offers early white papers that Mr. Bejtlich wrote as a network defender, either for technica...
Read more