Monday, November 22, 2010

Global Policies - Squid Proxy

Hey There,

Today I was thrown with this task of making sure that all our proxies in my company have the same White and the Black List. The company already has the List of category that they block, with a standard squid.conf and sfagent, Now they want to go ahead and block a website or allow it in less than 30 minutes all over the company. If it is small number of proxies then I can guess this can be manual, but in our company, we have greater than 150 proxies which are a mix of Bluecoat and squid. Bluecoats, it was very easy by using the central policy concept, for the squids, I decided that we should create a script that goes ahead and makes changes the squid.conf file adds a crontab, so on and so forth.

So I used the same Webserver, which the company was  using for the Bluecoat central policies and made a blocked list and allowed list text files. Then I created a shell script installer, which when run in our company environment, will modify the squid.conf file adding references of the allowed and the blocked list and also insert a crontab

I wrote the script in haste, so there might be a lot of bugs, but for the people it might be interesting, I will develop this as a full squid addon and put this on GNU, for the people who know shell, it might not be a great deal.

Also, to use the script, go ahead and copy this in you squid proxies (change the to  your webserver ip and set the paths) hope this atleast gives an idea to the people

# Author: Alok. A. S
# Global Policy Poller for Squid Proxies on Linux
# This will use the WGET installed on the system
# You will also need to run the ./ install to get instructions

if [ "$1" == "install" ]
        echo ""
        echo "********************************************************************"
        echo " "
        echo "      Welcome to the Global Policy Script Installer Function        "
        echo "   This script function will install itself into the configuration  "
        echo "       folder, you will have to put the script in crontab           "
        echo "  "
        echo "         Author : Alok. A. S (                "
        echo "            Please direct any bug reports to the Author             "
        echo ""
        echo "********************************************************************"
        echo "Starting ..."
        echo ""

        os=`uname -a | awk '{print $1}'`
        wget_check=`which wget`
        squid_check=`which squid`
        config_file=`find / -name squid.conf | grep etc | head -1`
        scriptname=`basename $0`
        scriptdirname=`dirname $0`

        check=`echo $os | grep x`
        if [ $? -ne 0 ]
                echo "Sorry, Not a *nix system, Will Not install"
                exit 127
        echo "Operating System : $os ... (Ok)"
        if [ ! -f "$wget_check" ]
                echo "WGET Not found, Will Not Install ..."
                exit 127
        echo "WGET Location: $wget_check ... (Ok)"
        if [ ! -f "$squid_check" ]
                echo "Squid executable not found, Will Not Install ..."
                exit 127
        echo "Squid Executable: $squid_check ... (Ok)"
        echo ""
        echo "Prerequisites checked fine, Continuing with Installation ..."
        echo ""
        echo ""
        echo "Squid Config File : $config_file"
        read -p  'Is this Correct? (y/n) ' option
        if [ $option != 'y' ]
                echo ""
                read -p "Sorry, my bad, can you please tell me the path and file name of the config file ? " config_file
                echo ""
                echo "Thanks, Proceeding ..."
        config_directory=`dirname $config_file`
        isinstalled=`cat $config_file | grep -v "#" | grep global_blocklist`
        if [ $? == 0 ]
                echo "Looks like it is already installed, Will exit"
                echo "The following line(s) was/were found : "
                echo ""
                echo "$isinstalled"
                echo ""
                exit 127
        lastline=`cat -n $config_file | tail -1 | awk '{print $1}'`
        firstacl=`cat -n $config_file | grep acl | head -1 | awk '{print $1}'`
        echo ""
        echo "Taking backup of original squid.conf (It will be appended with beforeglobal extension)"
        cp $config_file $config_file.`date +"%Y%m%d"`.beforeglobal
        echo ""
        echo "Creating a New Config file ..."
        newfile=$config_file.`date +"%Y%m%d"`.newconfig
        rm -rf $newfile
        touch $newfile
        chmod 666 $newfile

        lineno=`expr "$firstacl" - 1`
        remain=`expr "$lastline" - "$lineno"`


        `head -$lineno $config_file >> $newfile`
        echo "acl global_blocklist url_regex \"$blocklist\"" >> $newfile
        echo "acl global_allowedlist url_regex \"$allowedlist\"" >> $newfile
        `tail -$remain $config_file >> $newfile`

        newfile1=$config_file.`date +"%Y%m%d"`.newconfig1
        rm -rf $newfile1
        touch $newfile1
        chmod 666 $newfile1
        lastline=`cat -n $newfile | tail -1 | awk '{print $1}'`
        firstacl=`cat -n $newfile | grep http_access | head -1 | awk '{print $1}'`

        lineno=`expr "$firstacl" - 1`
        remain=`expr "$lastline" - "$lineno"`

        `head -$lineno $newfile >> $newfile1`
        echo "http_access deny global_blocklist all" >> $newfile1
        echo "http_access allow global_allowedlist all" >> $newfile1
        `tail -$remain $newfile >> $newfile1`

        rm -rf $newfile
        read -p 'Can i replace the squid.conf file with the new one ? (y/n)  ' option
        if [ $option != 'y' ]
                echo ""
                echo "Ok, Please do it manually, Move the script to the $config_directory and dont forget to add the crontab entry ..."
                echo "Execute squid -k reconfigure for this to take effect"
                echo ""
                echo "To Add a crontab, type the command crontab -e"
                echo "Then add the following line (in the last line) "
                echo "0,20,40 * * * * $config_directory/$scriptname"
                echo ""
                echo "and exit with wq! (like VI)"
                echo "Ciao ... "
                echo ""
                exit 127
        rm -rf $config_file
        mv $newfile1 $config_file
        echo ""
        echo "The new config file is in place ... Proceeding ..."
        cp -f $myfilename $myconfigdir
        rm -rf "$scriptdirname/temcrontabfile.txt"
        touch "$scriptdirname/temcrontabfile.txt"
        `crontab -l | grep -v "#" >> "$scriptdirname/temcrontabfile.txt"`
        echo "0,20,40 * * * * $config_directory/$scriptname" >> "$scriptdirname/temcrontabfile.txt"
        `crontab "$scriptdirname/temcrontabfile.txt"`
        rm -rf "$scriptdirname/temcrontabfile.txt"
        echo "Crontab Installed ..."
        echo "Script copied to the squid folder ... "
        echo "Reconfiguring Squid ... "
        `$squid_check -k reconfigure`
        echo "The installation completed, the global list will be downloaded after 20 mins (or) run the script without the install option in $config_directory/"
        echo "Thanks ... "
        exit 127

scriptname=`basename $0`
scriptdirname=`dirname $0`


if [ ! -f "$logfile" ]
    touch $logfile
    chmod 666 $logfile
    # echo "File created"

if [ ! -f "$blockedlist" ]
        echo "$blockedlist not found, Creating it ..."
        touch $blockedlist
        chmod 666 $blockedlist

if [ ! -f "$allowedlist" ]
        echo "$allowedlist not found, Creating it ..."
        touch $allowedlist
        chmod 666 $allowedlist

blockmd5=`md5sum $blockedlist | awk '{print $1}'`
allowmd5=`md5sum $allowedlist | awk '{print $1}'`

wget_check=`which wget`

newblockedlist="$scriptdirname/newblockedlist".`date +"%Y%m%d"`
newallowedlist="$scriptdirname/newallowedlist".`date +"%Y%m%d"`


$wget_check -q $urlblock -O $newblockedlist -T 5
$wget_check -q $urlallow -O $newallowedlist -T 5

if [ ! -f "$newblockedlist" ]
        echo "$newblockedlist not found, not downloaded, so exiting ..."
        exit 127

if [ ! -f "$newallowedlist" ]
        echo "$newallowedlist not found, not downloaded, so exiting ..."
        exit 127

blockmd5new=`md5sum $newblockedlist | awk '{print $1}'`
allowmd5new=`md5sum $newallowedlist | awk '{print $1}'`

if [ "$blockmd5new" != "$blockmd5" ]
        mv "$blockedlist" "$blockedlist.backup".`date +"%Y%m%d"`
        mv "$newblockedlist" "$blockedlist"
        echo `date` "Blocked List Updated" >> $logfile

if [ "$allowmd5new" != "$allowmd5" ]
        mv "$allowedlist" "$allowedlist.backup".`date +"%Y%m%d"`
        mv "$newallowedlist" "$allowedlist"
        echo `date` "Allowed List Updated" >> $logfile

rm -rf "$newallowedlist"
rm -rf "$newblockedlist"

You need to copy it run the script with ./ install, and then it will do its thing. Hope this helps ....

How does a Simple Browser Works ... Basics

Hi There,

Its been a long time since i have posted something here, owing to the fact that I was busy in the monotonous work.

Anycase, todays topic is how the browser works, I am amazed at how many people take it for granted and don't really understand the basics behind it,

Like the way I like it, lets start with some practical stuff then I will explain what we did in the exercise.

This is what you need , (Assuming you are reading this, you will have them)

1. PC connected to the Internet directly, without using a Explicit proxy (Transparent Proxies are ok),

How to Check:

On Windows:
If you don't know if your PC is proxied or not, a good way to check is your Internet Explorer, Go to Tools --> Internet Options --> Connections

Click on the LAN Settings button, Make sure there is nothing checked, if you are able to browse internet, means you either have no proxy or a transparent proxy (Both are OK)

On Linux:
If you use Linux, you might already know how to do it ... But here is the way anyway, get into the terminal and type the command

echo $http_proxy

2. A command line agent which has telnet enabled, If you are on windows vista or 7, you may not have it.

How to check:

On Windows,

Start --> Run
Type the command "cmd" (Without quotes) The command prompt opens, type telnet and hit the enter button, if you get a "Microsoft Telnet" then you are fine

All the linux terminals have telnet clients (Unless you explicitly removed it), so if you running linux, you are ok

In windows Vista and 7, here is how you install it, Go to Control Panel --> Add remove Programs, Click on the Add remove Windows Components and select the Telnet client and click on Ok, it will install the client for you

Well, Thats it .... for HTTP command line testing you need thats all

Lets start with a simple google page download trick, so you see what your browser does

Open the terminal

type the following

telnet 80

Now if you get a Blank screen in windows (and the prompt will request for more input in linux)

type the following

GET / HTTP/1.1
Connection: Close
(Hit Enter 2 times)

Also, if you are on Windows, you wont see what you are typing, thats ok ... Once you do, the terminal will spit html codes on your screen... Cool!!!

Now let us see what we did... We mimic the browser in the way it works.

OK, Now having seen the practical example, lets go to the theory how it works and before I start, I also want you to notice one more thing, when you type the "" in the browser, see what it becomes.

It changes into (Please note that in this whole example I am using the and not the .com, because we haven't spoken about redirection yet)

this full thing ( is called the URL or the Uniform Resource Locator, This address has 3 parts to it.

http:// -- This is the protocol that the resource needs to be accessed on -- Host name of the resource
/ -- The Trailing "/" is the URI (Uniform Resource Identifier)

If you don't mention anything, it will automatically append the "/" as that symbolizes the root directory of the web server.

The first part is important, we have different ways to access resources like,

http:// (Unencrypted Hyper Text Transfer Protocol)
https:// (Secure HTTP - Encrypted)
ftp:// (File Transfer Protocol)
ftps:// (Secure FTP)
telnet:// (Tel Networking)

so on and so forth. The most used ones by an average Internet user are the the first 3 in the line.

Each protocol has its own access methods, (Mental Note : STOP Digressing)

As far as HTTP is concerned, there are 2 important methods GET and POST. GET is the most used, for fetching of web pages. POST is used to, well ... POST variables out to the web server (Like Credit Card Information when you are purchasing some thing, etc)

ok, having said this, Lets analyse what we did first ...

"telnet 80", In this we just connected to the Web Server on port 80. Even there was some fundamental behind it, the is referenced on the internet by an IP address, so the telnet program, went ahead and did a DNS lookup and found the IP address and then connected to the IP on port 80 (which is the default http port)

Once it was connected, it waited for us to give some commands to it, Now since we have already connected to the google server, all we need to do is ask for some thing and that is what we did in the next line

We said

GET / HTTP/1.1

This means, we said, GET me the "/" page and use the HTTP1.1 Standard (More on this on some other post)

Next we said,


Where we set our header that we said, we are trying to access for server (This was not needed in the HTTP1.0 standard, but more on that later)

Then we said

Connection: Close

Meaning, please close the connection to the server after you have downloaded the page. The Server then goes ahead and gives us the HTML code , with a header.

Our browser renders that html and shows us the beautiful pages that we see.

So, these are the steps that a browser does in the back ground when we hit one site, not to go through the links and download pictures and stuff.

In practicality, here is what happens

1. Browser checks if there is a Proxy Server (If No Proceed Below steps else go to the proxy steps)
2. Do a DNS lookup on the Name and get an IP address
3. Connect to the IP on the port 80
4. Request a page
5. Display the page with its components

(Its actually more complicated than this, but this is the basic ides), so if the internet is not working, how would you test ... do the same thing in the local order

Note: If using the Proxy server, then this wont apply

Step1 : Check if the Computer has an IP addresss

go to the command prompt and type the command ipconfig /all

Make sure you have a VALID IP, not 169.x or 127.x
Make sure you have DNS server IP address

Example o/p
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Dell Wireless 1397 WLAN Mini-Card
Physical Address. . . . . . . . . : F0-7B-CB-89-14-CC
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . :
Subnet Mask . . . . . . . . . . . :
Default Gateway . . . . . . . . . :
DHCP Server . . . . . . . . . . . :
DNS Servers . . . . . . . . . . . :
Lease Obtained. . . . . . . . . . : Sunday, November 21, 2010 6:46:27 PM

Lease Expires . . . . . . . . . . : Monday, November 22, 2010 6:46:27 PM

You can see I have an IP address, if this is a problem, then the problem is with DHCP server or Static IP

Ok, now that out of the way, check if the DNS Server is working

from the command prompt, execute the command


You should get an IP address of

Sample O/p
C:\> nslookup

Non-authoritative answer:

In this case, your IE should be able to get to the site, if still not happening, do the telnet test, if that works, there is some problem with IE settings.

Well, I assume, you understood, the basics behind how a simple website is pulled by our browser.

Next Post ... HTTPS Troubleshooting ... Advanced