December 12, 2019 Python Networking : HTTP Programming

The ultimate security is your understanding of reality.
– H. Stanley Judd

This article will introduce the basics of Python HTTP programming. So far in this series, we have discussed Python security programming and Python network programming. In this article the reader will learn how to implement advanced Python HTTP-related scenarios and use cases. 

A web server is a computer server which stores content related to the web.  Websites are hosted on this type of server. Web servers are used for other purposes such as FTP, email, gaming, storage and more. Server is typically a term used for hardware and software. Hardware based web servers handle HTML pages, images, style sheets, javascript, and web server software. The server can handle data exchange with devices connected to the internet.

Web server software consists of user management, user access, and a server based on HTTP protocol. HTTP based servers can parse URLs which are web addresses. HTTP is the protocol used for browsing and rendering web content. Web sites are browsed by searching through domain names of the web servers. The web servers store web content and have domain names configured for the network DNS ip address.

A website has a group of web pages serving images, videos and audio content. An HTTP request is created by the browser to request web content on the web server. The web server returns the HTTP response by sharing the requested content with the URL. The response is generated by invoking a server side component which interacts with the database.

A web server is called static if the content is shared as it is available on the server. The examples are images, videos and audio files. On the other hand, a dynamic web server has a static module and a dynamic module rendering the response from the server side components. The architectural approaches used for a web server are concurrent and single process event driven architectural patterns.

A concurrent approach can process many HTTP requests simultaneously. This can be achieved by web servers with multiple threads, processes, and hybrid approaches of processes and threads. A simple web server creates a single thread. The single thread spawns multiple child processes for HTTP requests. The child processes handle the HTTP requests for further processing required for rendering the response. Multiple Processed based web servers have a master slave pattern based implementation. In this implementation, the master process manages the load by having child processes forked. Multiple Thread based servers manage multiple threads to handle concurrent requests. The hybrid approach is to have multiple processes created and threads being spawned by each process.

The diagram below shows the request and response mechanism implemented in the web server.

The web server listens to HTTP requests on a TCP IP Address and a port number. It responds to the HTTP request by sending the HTTP response. The HTTP headers have the source and destination information for requests and responses. A web server is yet another network server which can handle requests and responses. The web server acts as a request dispatcher and handlers are provided to render the response.

The code samples below show advanced Python HTTP scenario implementation. In this implementation, we look at the basic example in which Python is used for an HTTP web server.

Prerequisites:
1. You need to set up Python3.5 to run the code samples below. You can download from this link.

Problem
Http.server and socket server module has the features to create a TCP server and handling of HTTPRequests. The features related to HTTPServer implementation and client connection are shown in the example below. 

The code snippet below shows the Python web server using HTTP API.

import http.server
import socketserver
 
HTTPPORT = 8000
HTTPRequestHandler = http.server.SimpleHTTPRequestHandler
 
with socketserver.TCPServer(("", HTTPPORT), HTTPRequestHandler) as httpd:
    print("web server started", HTTPPORT)
    httpd.serve_forever()

HTTPRequestHandler of http.server module  is used to handle the requests and serve the files from the directory of the web server. TCPServer of socket server module is used to create an HTTP web server httpd. This server receives and sends httprequests and httpresponses respectively using TCP protocol. HTTP is a TCP protocol. It is an application layer protocol.

The web server is instantiated with an IP address and port number. The handler is passed, which is a Simple HTTPRequestHandler. serve_forever method is on httpd. This method is used to start the web server. The web server receives the requests  and sends the responses which are HTTP protocol messages.

HTTP webserver is started by executing the commands shown below.

Instructions for Running the Code

#running the python web server
python3 web_server.py

Output

Further in this series, we are going to cover network traffic analysis, information gathering from servers, FTP, SSH & SNMP interaction, cryptography, nmap scanning, geolocation & metadata retrieval, and  vulnerability scanning and analysis.