datavow.com - Documents

1) Introduction
- 1) Purpose
- 2) Requirements
- 3) Terminology
- 4) Overall Operation
2) Notational Conventions and Generic Grammar
- 1) Augmented BNF
- 2) Basic Rules
3) Protocol Parameters
- 1) HTTP Version
- 2) Uniform Resource Identifiers
- 3) Date/Time Formats
  - 1) Full Date
  - 2) Delta Seconds
- 4) Character Sets
  - 1) Missing Charset
- 5) Content Codings
- 6) Transfer Codings
  - 1) Chunked Transfer Coding
- 7) Media Types
  - 1) Canonicalization and Text Defaults
  - 2) Multipart Types
- 8) Product Tokens
- 9) Quality Values
- 10) Language Tags
- 11) Entity Tags
- 12) Range Units
4) HTTP Message
- 1) Message Types
- 2) Message Headers
- 3) Message Body
- 4) Message Length
- 5) General Header Fields
5) Request
- 1) Request-Line
  - 1) Method
  - 2) Request-URI
- 2) The Resource Identified by a Request
- 3) Request Header Fields
6) Response
- 1) Status-Line
  - 1) Status Code and Reason Phrase
- 2) Response Header Fields
7) Entity
- 1) Entity Header Fields
- 2) Entity Body
  - 1) Type
  - 2) Entity Length
8) Connections
- 1) Persistent Connections
- 2) Message Transmission Requirements
9) Method Definitions
- 1) Safe and Idempotent Methods
  - 1) Safe Methods
  - 2) Idempotent Methods
- 2) OPTIONS
- 3) GET
- 4) HEAD
- 5) POST
- 6) PUT
- 7) DELETE
- 8) TRACE
- 9) CONNECT
10) Status Code Definitions
- 1) Informational 1xx
  - 1) Continue
  - 2) Switching Protocols
- 2) Successful 2xx
- 3) Redirection 3xx
- 4) Client Error 4xx
- 5) Server Error 5xx
11) Access Authentication
12) Content Negotiation
- 1) Server-driven Negotiation
- 2) Agent-driven Negotiation
- 3) Transparent Negotiation
13) Caching in HTTP
- 1) ..
- 2) Expiration Model
- 3) Validation Model
- 4) Response Cacheability
- 5) Constructing Responses From Caches
- 6) Caching Negotiated Responses
- 7) Shared and Non-Shared Caches
- 8) Errors or Incomplete Response Cache Behavior
- 9) Side Effects of GET and HEAD
- 10) Invalidation After Updates or Deletions
- 11) Write-Through Mandatory
- 12) Cache Replacement
- 13) History Lists
14) Header Field Definitions
- 1) Accept
- 2) Accept-Charset
- 3) Accept-Encoding
- 4) Accept-Language
- 5) Accept-Ranges
- 6) Age
- 7) Allow
- 8) Authorization
- 9) Cache-Control
  - 1) What is Cacheable
  - 2) What May be Stored by Caches
  - 3) Modifications of the Basic Expiration Mechanism
  - 4) Cache Revalidation and Reload Controls
  - 5) No-Transform Directive
  - 6) Cache Control Extensions
- 10) Connection
- 11) Content-Encoding
- 12) Content-Language
- 13) Content-Length
- 14) Content-Location
- 15) Content-MD5
- 16) Content-Range
- 17) Content-Type
- 18) Date
  - 1) Clockless Origin Server Operation
- 19) ETag
- 20) Expect
- 21) Expires
- 22) From
- 23) Host
- 24) If-Match
- 25) If-Modified-Since
- 26) If-None-Match
- 27) If-Range
- 28) If-Unmodified-Since
- 29) Last-Modified
- 30) Location
- 31) Max-Forwards
- 32) Pragma
- 33) Proxy-Authenticate
- 34) Proxy-Authorization
- 35) Range
  - 1) Byte Ranges
  - 2) Range Retrieval Requests
- 36) Referer
- 37) Retry-After
- 38) Server
- 39) TE
- 40) Trailer
- 41) Transfer-Encoding
- 42) Upgrade
- 43) User-Agent
- 44) Vary
- 45) Via
- 46) Warning
- 47) WWW-Authenticate
15) Security Considerations
- 1) Personal Information
- 2) Attacks Based On File and Path Names
- 3) DNS Spoofing
- 4) Location Headers and Spoofing
- 5) Content-Disposition Issues
- 6) Authentication Credentials and Idle Clients
- 7) Proxies and Caching
  - 1) Denial of Service Attacks on Proxies
16) Acknowledgments
17) References
18) Authors' Addresses
19) Appendices
- 1) Internet Media Type message/http and application/http
- 2) Internet Media Type multipart/byteranges
- 3) Tolerant Applications
- 4) Differences Between HTTP Entities and RFC 2045 Entities
- 5) Additional Features
  - 1) Content-Disposition
- 6) Compatibility with Previous Versions
20) Index
21) Full Copyright Statement
22) Acknowledgement

8 Connections

8.1 Persistent Connections

8.1.1 Purpose

Prior to persistent connections, a separate TCP connection was established to fetch each URL, increasing the load on HTTP servers and causing congestion on the Internet. The use of inline images and other associated data often require a client to make multiple requests of the same server in a short amount of time. Analysis of these performance problems and results from a prototype implementation are available [26] [30]. Implementation experience and measurements of actual HTTP/1.1 (RFC 2068) implementations show good results [39]. Alternatives have also been explored, for example, T/TCP [27].

Persistent HTTP connections have a number of advantages:

- By opening and closing fewer TCP connections, CPU time is saved in routers and hosts (clients, servers, proxies, gateways, tunnels, or caches), and memory used for TCP protocol control blocks can be saved in hosts.

- HTTP requests and responses can be pipelined on a connection. Pipelining allows a client to make multiple requests without waiting for each response, allowing a single TCP connection to be used much more efficiently, with much lower elapsed time.

- Network congestion is reduced by reducing the number of packets caused by TCP opens, and by allowing TCP sufficient time to determine the congestion state of the network.

- Latency on subsequent requests is reduced since there is no time spent in TCP's connection opening handshake.

HTTP can evolve more gracefully, since errors can be reported without the penalty of closing the TCP connection. Clients using future versions of HTTP might optimistically try a new feature, but if communicating with an older server, retry with old semantics after an error is reported.

HTTP implementations SHOULD implement persistent connections.

8.1.2 Overall Operation

A significant difference between HTTP/1.1 and earlier versions of HTTP is that persistent connections are the default behavior of any HTTP connection. That is, unless otherwise indicated, the client SHOULD assume that the server will maintain a persistent connection, even after error responses from the server.

Persistent connections provide a mechanism by which a client and a server can signal the close of a TCP connection. This signaling takes place using the Connection header field (Section 14.10). Once a close has been signaled, the client MUST NOT send any more requests on that connection.

8.1.2.1 Negotiation

An HTTP/1.1 server MAY assume that a HTTP/1.1 client intends to maintain a persistent connection unless a Connection header including the connection-token "close" was sent in the request. If the server chooses to close the connection immediately after sending the response, it SHOULD send a Connection header including the connection-token close.

An HTTP/1.1 client MAY expect a connection to remain open, but would decide to keep it open based on whether the response from a server contains a Connection header with the connection-token close. In case the client does not want to maintain a connection for more than that request, it SHOULD send a Connection header including the connection-token close.

If either the client or the server sends the close token in the Connection header, that request becomes the last one for the connection.

Clients and servers SHOULD NOT assume that a persistent connection is maintained for HTTP versions less than 1.1 unless it is explicitly signaled. See Section 19.6.2 for more information on backward compatibility with HTTP/1.0 clients.

In order to remain persistent, all messages on the connection MUST have a self-defined message length (i.e., one not defined by closure of the connection), as described in Section 4.4.

8.1.2.2 Pipelining

A client that supports persistent connections MAY "pipeline" its requests (i.e., send multiple requests without waiting for each response). A server MUST send its responses to those requests in the same order that the requests were received.

Clients which assume persistent connections and pipeline immediately after connection establishment SHOULD be prepared to retry their connection if the first pipelined attempt fails. If a client does such a retry, it MUST NOT pipeline before it knows the connection is persistent. Clients MUST also be prepared to resend their requests if the server closes the connection before sending all of the corresponding responses.

Clients SHOULD NOT pipeline requests using non-idempotent methods or non-idempotent sequences of methods (see Section 9.1.2). Otherwise, a premature termination of the transport connection could lead to indeterminate results. A client wishing to send a non-idempotent request SHOULD wait to send that request until it has received the response status for the previous request.

8.1.3 Proxy Servers

It is especially important that proxies correctly implement the properties of the Connection header field as specified in Section 14.10.

The proxy server MUST signal persistent connections separately with its clients and the origin servers (or other proxy servers) that it connects to. Each persistent connection applies to only one transport link.

A proxy server MUST NOT establish a HTTP/1.1 persistent connection with an HTTP/1.0 client (but see RFC 2068 [33] for information and discussion of the problems with the Keep-Alive header implemented by many HTTP/1.0 clients).

8.1.4 Practical Considerations

Servers will usually have some time-out value beyond which they will no longer maintain an inactive connection. Proxy servers might make this a higher value since it is likely that the client will be making more connections through the same server. The use of persistent connections places no requirements on the length (or existence) of this time-out for either the client or the server.

When a client or server wishes to time-out it SHOULD issue a graceful close on the transport connection. Clients and servers SHOULD both constantly watch for the other side of the transport close, and respond to it as appropriate. If a client or server does not detect the other side's close promptly it could cause unnecessary resource drain on the network.

A client, server, or proxy MAY close the transport connection at any time. For example, a client might have started to send a new request at the same time that the server has decided to close the "idle" connection. From the server's point of view, the connection is being closed while it was idle, but from the client's point of view, a request is in progress.

This means that clients, servers, and proxies MUST be able to recover from asynchronous close events. Client software SHOULD reopen the transport connection and retransmit the aborted sequence of requests without user interaction so long as the request sequence is idempotent (see Section 9.1.2). Non-idempotent methods or sequences MUST NOT be automatically retried, although user agents MAY offer a human operator the choice of retrying the request(s). Confirmation by user-agent software with semantic understanding of the application MAY substitute for user confirmation. The automatic retry SHOULD NOT be repeated if the second sequence of requests fails.

Servers SHOULD always respond to at least one request per connection, if at all possible. Servers SHOULD NOT close a connection in the middle of transmitting a response, unless a network or client failure is suspected.

Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than 2 connections with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion.

8.2 Message Transmission Requirements

8.2.1 Persistent Connections and Flow Control

HTTP/1.1 servers SHOULD maintain persistent connections and use TCP's flow control mechanisms to resolve temporary overloads, rather than terminating connections with the expectation that clients will retry. The latter technique can exacerbate network congestion.

8.2.2 Monitoring Connections for Error Status Messages

An HTTP/1.1 (or later) client sending a message-body SHOULD monitor the network connection for an error status while it is transmitting the request. If the client sees an error status, it SHOULD immediately cease transmitting the body. If the body is being sent using a "chunked" encoding (Section 3.6), a zero length chunk and empty trailer MAY be used to prematurely mark the end of the message. If the body was preceded by a Content-Length header, the client MUST close the connection.

8.2.3 Use of the 100 (Continue) Status

The purpose of the 100 (Continue) status (see Section 10.1.1) is to allow a client that is sending a request message with a request body to determine if the origin server is willing to accept the request (based on the request headers) before the client sends the request body. In some cases, it might either be inappropriate or highly inefficient for the client to send the body if the server will reject the message without looking at the body.

Requirements for HTTP/1.1 clients:

- If a client will wait for a 100 (Continue) response before sending the request body, it MUST send an Expect request-header field (Section 14.20) with the "100-continue" expectation.

- A client MUST NOT send an Expect request-header field (Section 14.20) with the "100-continue" expectation if it does not intend to send a request body.

Because of the presence of older implementations, the protocol allows ambiguous situations in which a client may send "Expect: 100- continue" without receiving either a 417 (Expectation Failed) status or a 100 (Continue) status. Therefore, when a client sends this header field to an origin server (possibly via a proxy) from which it has never seen a 100 (Continue) status, the client SHOULD NOT wait for an indefinite period before sending the request body.

Requirements for HTTP/1.1 origin servers:

- Upon receiving a request which includes an Expect request-header field with the "100-continue" expectation, an origin server MUST either respond with 100 (Continue) status and continue to read from the input stream, or respond with a final status code. The origin server MUST NOT wait for the request body before sending the 100 (Continue) response. If it responds with a final status code, it MAY close the transport connection or it MAY continue to read and discard the rest of the request. It MUST NOT perform the requested method if it returns a final status code.

- An origin server SHOULD NOT send a 100 (Continue) response if the request message does not include an Expect request-header field with the "100-continue" expectation, and MUST NOT send a 100 (Continue) response if such a request comes from an HTTP/1.0 (or earlier) client. There is an exception to this rule: for compatibility with RFC 2068, a server MAY send a 100 (Continue) status in response to an HTTP/1.1 PUT or POST request that does not include an Expect request-header field with the "100- continue" expectation. This exception, the purpose of which is to minimize any client processing delays associated with an undeclared wait for 100 (Continue) status, applies only to HTTP/1.1 requests, and not to requests with any other HTTP- version value.

- An origin server MAY omit a 100 (Continue) response if it has already received some or all of the request body for the corresponding request.

- An origin server that sends a 100 (Continue) response MUST ultimately send a final status code, once the request body is received and processed, unless it terminates the transport connection prematurely.

- If an origin server receives a request that does not include an Expect request-header field with the "100-continue" expectation, the request includes a request body, and the server responds with a final status code before reading the entire request body from the transport connection, then the server SHOULD NOT close the transport connection until it has read the entire request, or until the client closes the connection. Otherwise, the client might not reliably receive the response message. However, this requirement is not be construed as preventing a server from defending itself against denial-of-service attacks, or from badly broken client implementations.

Requirements for HTTP/1.1 proxies:

- If a proxy receives a request that includes an Expect request- header field with the "100-continue" expectation, and the proxy either knows that the next-hop server complies with HTTP/1.1 or higher, or does not know the HTTP version of the next-hop server, it MUST forward the request, including the Expect header field.

- If the proxy knows that the version of the next-hop server is HTTP/1.0 or lower, it MUST NOT forward the request, and it MUST respond with a 417 (Expectation Failed) status.

- Proxies SHOULD maintain a cache recording the HTTP version numbers received from recently-referenced next-hop servers.

- A proxy MUST NOT forward a 100 (Continue) response if the request message was received from an HTTP/1.0 (or earlier) client and did not include an Expect request-header field with the "100-continue" expectation. This requirement overrides the general rule for forwarding of 1xx responses (see Section 10.1).

8.2.4 Client Behavior if Server Prematurely Closes Connection

If an HTTP/1.1 client sends a request which includes a request body, but which does not include an Expect request-header field with the "100-continue" expectation, and if the client is not directly connected to an HTTP/1.1 origin server, and if the client sees the connection close before receiving any status from the server, the client SHOULD retry the request. If the client does retry this request, it MAY use the following "binary exponential backoff" algorithm to be assured of obtaining a reliable response:

Initiate a new connection to the server
Transmit the request-headers
Initialize a variable R to the estimated round-trip time to the server (e.g., based on the time it took to establish the connection), or to a constant value of 5 seconds if the round- trip time is not available.
Compute T = R * (2**N), where N is the number of previous retries of this request.
Wait either for an error response from the server, or for T seconds (whichever comes first)
If no error response is received, after T seconds transmit the body of the request.
If client sees that the connection is closed prematurely, repeat from step 1 until the request is accepted, an error response is received, or the user becomes impatient and terminates the retry process.

If at any point an error status is received, the client

- SHOULD NOT continue and

- SHOULD close the connection if it has not completed sending the request message.