Common Gateway Interface (CGI) Programming

CGI was the first standard means of creating dynamic web content. Dynamic web content refers to The term "CGI programming" is sort of a misnomer, since CGI is simply a means of doing the programming; a programming or scripting language is necessary for an actual app. CGI is simply the means of communication through the web server.

CGI programs are stored in a directory which must be configured in the web server.  The path is typically SERVER_ROOT/cgi-bin, so the URL looks like

When the client requests a URL to a CGI script the server doesn't return the contents of the file, rather it executes the script and sends the output produced by the script back to the client.  The output produced by the script includes two main chunks: The language used in CGI programming is not specified.  You want something that can: Choices include: On UNIX any program will work which reads/writes its stdin/stdout.

Of these, perl is very highly suited since it has powerful text manipulation built-in, you can call any C library function or UNIX utility, and it's easy to get to the environmental variables.  This has made perl the most widely used language for CGI programming.  Perl is also available for UNIX, Macintosh and NT.

Environmental Variables

Lots of info is passed this way:
 <slide of environmental variables>

Since you have all this info about the client and user you can do things like:

<simple perl example slide>

Form Input

<Figure 7-67 Tanenbaum>

Output

The point of the CGI program is to produce something that the browser can display.  Possibilities are: The CGI program may send either a partial or complete HTTP header.  If partial, the web server will complete.  Here's what can be in a header: The header block ends in a blank line, which your CGI program must produce.

The client lets the CGI program know what types it will accept via the HTTP_ACCEPT variable.  The data sent back via Content-type should be one of these types, or the browser won't know what to do with it.

Server redirection

When a CGI program just wants to send a file back, say a GIF image, it doesn't need to read/write the contents of the file.  It can much more efficiently do this by re-directing the request to a file or URL.

<example of load-dependent response>

Advanced CGI Programming

Server side includes (SSI)

Motivation for SSI: a document in which 95% of content is static, 5% dynamic.

Server must be capable, and must be configured to do SSI.

SSI gives you a means of including small amounts of dynamically generated data into a largely static document. The kind of info you can embed includes:

With SSI, the HTTP server must parse your file looking for the SSI directives.  This can be very expensive for large documents, and has the potential of bogging the server down, since the server's normal duties only include sending an HTML file to a client, and not interpreting it.

Simple example:

<html>
<head>
 <title>
 </title>
</head>
<body>
Hello, you are from <!-- #echo var="SERVER_NAME"-->
</body>
</html>

A handy use for SSI is to include a "boilerplate" file.  This solves a problem of not being able to re-use common components in HTML documents without copying them.  A signature file is a good example:

Store the following in address.shtml:

<address>
<pre>
Your Name
Address
Email
Phone number
</pre>
</address>

Now you can include this file in any document via
 <!-- #include virtual="address.html"-->
 
The output of an external program (shell script, utility, etc) can be included via the SSI exec directive.  This can be a security hole, since a user may make a command's output available to the world which reveals sensitive data (like the passwd file).

Another problem with the exec directive is the possibility that someone enters HTML (e.g. via a guestbook CGI program) which contains SSI directives.  The next time this document is accessed the SSI directive entered by the user is executed.  This could include a "rm -rf" type command".

Among the external programs that can be executed are CGI programs.  This is how most access counters are done.

Server side includes can not be placed in CGI programs.

Dynamic creation of graphics

Several means are available to create graphics dynamically.  All involve exeuction of a CGI program.

If a PostScript interpreter is available on the web server, then a CGI program can produce PostScript code to create an image, interpret the PS, then convert to GIF or JPEG.  The GNU GhostScript package does this nicely and is widely available.

Another common library is called "gd".  It lets you make simple graphic images and output as GIF.  The gd library is available from perl, tcl and C.

For plotting data, the gnuplot package is handy.  I've seen this used on one of the stock market quote sites.

Forcing updates

What if you want to have a client automatically load/re-load a document?  Normally this is user driven, but some applications, such as displaying the current value of a stock every 15 minutes, or doing graphical animations, require automation of loading. The underlying problem is that automatic updates breaks the assumed pure client/server model.  Hacks are necessary.

Two possibilities: client pull, server push.

Client pull

The browser is tricked into re-loading a page via the Refresh HTTP header.  Here's  and example that will display the local time (on the server) on the browser every 5 seconds.  The time is gotten from an SSI directive.
 
 <meta http-equiv="Refresh" content=5>
 <!--#echo var="DATE_LOCAL"-->

Server push

A MIME type (multipart/x-mixed-replace) is available which causes the http server to send multiple packets, separated by a time delay.  The browser (assuming it knows this extension) replaces the contents of the previous packet with the contents of the latest packet.  This allow for animations.

Animation, which is probably the most common use of these techniques, can be done much more cleanly by making it the responsibility of the client, not the server.

A better means of doing animation is to use the feature of the gif89 standard which includes multiple images and the display loop.

One way of making the client smart is to distribute Java applets.

Preserving state information

The underlying model for HTTP is stateless.  In other words, all HTTP transactions are independent of each other.  Each opens a new connection, makes a request, gets a response, and breaks the connection.  Nothing is rememberd by the server once a transaction is completed.

What if you want to do something that requires maintaining state in the server?  A need like this arises if you want to do a "shopping cart" type web site, where users build up a list of stuff to buy.  Each addition to the list is a new transaction, yet the web server must be able to associate many transactions in a row with each other.  A hack is necessary.

The users view of the need to maintain state is when she interacts with multiple forms that are linked together via the semantics of the site.

Three possible strategies:

Hidden fields

Hidden fields are not secret.  They simply don't get displayed by the browser.

The first form gathers data with a form.  The CGI program then takes this data and makes it a part of the next HTML document sent to the user via a hidden form field.  The HTML documents keep getting longer and longer as data is built up.

CGI side includes

Instead of placing an HTML document inside a CGI program to be dynamically generated, you can put a CGI include directive in an HTML document and have a CGI program parse them out and replace them.

Persistent cookies

From Netscape.  The browser stores information that can be gotten at by the CGI program because the browser passes the information in a HTTP_COOKIE environmental variable.

The "cookies" are set to things like random numbers combined with host IP addresses.  This is done to ensure that all clients are uniquely identified.  When a client contacts a CGI program on the server it passes the magic cookie and the CGI program can act according to state information it has saved for this client.

Cookies have the following format

There is a fairly low limit to the number of cookies preserved by a client, and the size of the information associated with them.  The CGI program must use the file system to preserve state information because it needs to preserve this info across process boundaries (i.e. multiple instantiations of the CGI program).

A better alternative would be to use a "cookie server" model.  The CGI program identfies a user connection with a single magic cookie number.  This number is handed out by a cookie server which the CGI program talks to (say via a socket). The cookie server can then store information for the CGI program according to the magic cookie.  The advantage of this scheme is that the cookie server is always running so the information stays in memory, which is must faster than the file system.

Security and CGI

(from the Web Security FAQ http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.html)

The types of problems are:

1. Private or confidential documents stored in the Web site's document tree falling into the hands of unauthorized individuals.
2. Private or confidential information sent by the remote user to the server (such as credit card information) being intercepted.
3. Information about the Web server's host machine leaking through, giving outsiders access to data that can potentially allow them to break into the host.
4. Bugs that allow outsiders to execute commands on the server's host machine, allowing them to modify and/or damage the system. This includes "denial of service" attacks, in which the attackers pummel the machine with so many requests that it is rendered effectively useless.

Summary on CGI

The best things that can be said about CGI is that it was the first standard means of creating dynamic web content, and that it didn't force a choice of programming language.

Some problems with CGI are: