Overview
The HTTP message format is widely used, not just for web pages but also as a common message format between computers across the Internet. At Inesonic, we have several specialized engines for both SpeedSentry and our flagship product, Aion, that must work directly with the HTTP 1.1 message format.
The HTTP message format, version 1.1, is defined by IETF RFC 7230. There is also the more complex and much newer HTTP 2.0 message format covered by IETF RFC 7540 which I will not cover here.
HTTP messages rely on reliable transmission of data provided by the Internet’s TCP layer protocol.
The HTTP 1.1 message format, at it’s core is broken down into three parts:
- The start line
- Header fields
- Message body
The same message format is used, with some variation for both the request and the response. A simple HTTP request message might look like the content below:
GET /mypage.html HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux; x86_64; rv:62.0) Gecko/20100101 Firefox/62.0
Host: www.inesonic.com
Accept-Language: en
Accept-Encoding: gzip, deflate
And the returned response may look like:
HTTP/1.1 200 OK
Date: Sat, 29 Jan 2020 01:17:14 GMT
Server: Apache/2.4.41 (Ubuntu)
Last-Modified: Sat, 20 Jan 2022 01:13:24 GMT
ETag: “d0-5d6ae42126e6c”
Accept-Ranges: bytes
Content-Length: 208
Vary: Accept-Encoding
Content-Type: text/html
<!DOCTYPE html>
<html>
<body>
<p>
“Perfection is achieved, not when there is nothing more to add,
but when there is nothing more to take away.”
</p>
<p>-Antoine de Saint-Exupery</p>
</body>
</html>
The Start Line
Both the request and the response contain a start line, shown in blue above, although the content and format of the start lines are different. The start line in the request is referred to as the Request Line. The start line in the response is referred to as the Status Line.
The Request Line
The request start line contains three distinct space separated fields:
- The HTTP method or verb.
- The request target
- The HTTP version.
The line must be terminated with the special characters carriage return + line feed, that is ASCII/Latin 1 character code 0x0D followed by character code 0x0A.
The HTTP Method or Verb
The HTTP method or verb is the command being issued to the website. There are many different HTTP methods. The most common are:
- GET
- HEAD
- POST
- PUT
- DELETE
- PATCH
- TRACE
- OPTIONS
Some HTTP methods forbid a body in the request or in the response. For more details, see HTTP Methods.
Inesonic SpeedSentry supports all of the HTTP methods listed above except “TRACE”. Please contact us if you need Inesonic SpeedSentry to support other HTTP methods.
The Request Target
The request target indicates the path or endpoint being acted on. If your site’s URL is https://www.inesonic.com/product/, the request target would be:
/product/
For typical browser functions, the request target simply indicates the page to be fetched or updated. For REST API commands, the request target represents the endpoint, the command, and often, parameters associated with the call.
The HTTP Version
The HTTP version entry should be set to HTTP/1.1 for messages conforming to IETF RFC 7230.
The Status Line
Like the Request Line, the status line contains three space separated fields although the field contents and purpose are different:
- The HTTP version following the same rules as with the Request Line.
- The status code
- The reason phrase
Like the Request Line, the status line must be terminated with a carriage return + line feed.
The Status Code
There are a large number of status codes broken down into 5 different classes:
| Range | Class |
| 100 – 199 | Informational |
| 200 – 299 | Success |
| 300 – 399 | Redirection |
| 400 – 499 | Client error |
| 500 – 599 | Server error |
The Reason Phrase
The reason phrase is a textual description of the status code and is included in keeping with older internet protocols that were used with interactive text clients. RFC 7230 indicates that the reason phrase, while required, should be ignored.
The Header Fields
The header fields, colored in orange above, provide metadata associated with the request or response. Header fields are defined as key-value pairs, separated by a colon (“:”). Any whitespace after the colon, before the value, should be ignored. Each header field is terminated with a carriage return followed by a line feed.
The keys are case insensitive so the key “User-Agent”, “user-agent”, and “uSeR-AGenT” are the same. Headers should be interpreted in the order presented and later header key-value pairs may override or modify how earlier key-value pairs are interpreted.
Some header fields are intended to supply lists of values. For these cases, the values can either be provided by multiple identical entries or as a comma separated list in the same entry.
There are a large number of standard and defacto standard headers in use. The header fields are also used by the server to insert and obtain cookies stored by a customer’s browser.
The table below provides a very incomplete lists of common headers:
| Header Key | Function |
| User-Agent | Indicates the type of device sending a the request |
| Host: | The name of the host system a request is being sent to. |
| From: | A header required for bots that should include a contact email address. |
| Accept-Language: | The language(s) the client should receive page data for. |
| Accept-Encoding: | Responses can optionally be compressed. This header indicates the compression algorithms the client can support. |
| Content-Encoding: | Responses can include this header to indicate the encoding used by the server. Value should be one of the encodings indicated by the Accept-Encoding header in the request. |
| Cookie: | Provides cookies in a responses that should be stored by the client (browser). |
| Content-Length: | Provides the length of the message body, in bytes. Note that the presence of this header field indicates that the message will include body content. |
| Transfer-Encoding: | Indicates if and how the body content is compressed. The presence of this header field indicates that the message will include body content. |
| Connection: | Indicates if the connection between the client and server should be left open or closed after this transaction completes. |
The IETF RFC 7230 standard allows HTTP messages to be forwarded by proxy servers or other mechanisms. In some cases, a forwarding server may insert or modify header fields in the message. Example cases may be insertion of a “Host” header field or modifying the value in the “Transfer-Encoding” field.
Inesonic SpeedSentry allows you to specify values for the User-Agent and Content-Type headers. We automatically calculate and include a Content-Length header and include a From header.
The Message Body
The message body, colored in black above, contains the actual payload data to be sent to the server or returned from the server to the client.
The presence of a message body in the request and response depends heavily on the HTTP method specified in the request. Note that some HTTP methods, such as DELETE are allowed to optionally include a message body in either the request or the response.
The IETF RFC 7230 standard describes a number of methods for encoding the message body content.
The IETF RFC 7230 standard also describes how messages can be sent in multiple chunks using the “chunked” transfer encoding. Chunked messages are sent as multiple independent payloads, each with their own header. Chunked payloads allow content to be sent even if the size of the payload is not known up-front.
When you setup an Inesonic SpeedSentry monitor, with an HTTP method that allows a request body, you can specify the raw HTTP body to be sent.
Post photo by Brett Sayles from Pexels
