diff options
author | Pauli <pauli@openssl.org> | 2022-06-14 11:28:58 +1000 |
---|---|---|
committer | Pauli <pauli@openssl.org> | 2022-09-13 21:22:25 +1000 |
commit | fabce8090c3ba49527d434a4621c660eedad2aaa (patch) | |
tree | 6c9412349e9ad8660940f758920edc921ffbe11c /doc/designs/quic-design | |
parent | 0f4be8a14a2bcb8a92cf78d94d157152c0a03d88 (diff) | |
download | openssl-fabce8090c3ba49527d434a4621c660eedad2aaa.tar.gz |
design: TX Packetiser
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18570)
Diffstat (limited to 'doc/designs/quic-design')
-rw-r--r-- | doc/designs/quic-design/tx-packetiser.md | 436 |
1 files changed, 436 insertions, 0 deletions
diff --git a/doc/designs/quic-design/tx-packetiser.md b/doc/designs/quic-design/tx-packetiser.md new file mode 100644 index 0000000000..8e0b4a3094 --- /dev/null +++ b/doc/designs/quic-design/tx-packetiser.md @@ -0,0 +1,436 @@ +TX Packetiser +============= + +This module creates frames from the application data obtained from +the application. It also receives CRYPTO frames from the TLS Handshake +Record Layer and ACK frames from the ACK Handling And Loss Detector +subsystem. + +The packetiser also deals with the flow and congestion controllers. + +Creation & Destruction +---------------------- + +```c +struct ossl_quic_tx_packetiser_st { + QUIC_CONNECTION *conn; +}; + +_owur typedef struct ossl_quic_tx_packetiser_st OSSL_QUIC_TX_PACKETISER; + +OSSL_QUIC_TX_PACKETISER ossl_quic_tx_packetiser_new(QUIC_CONNECTION *conn); +void ossl_quic_tx_packetiser_free(OSSL_QUIC_TX_PACKETISER *tx); +``` + +Structures +---------- + +### Connection + +Represented by an QUIC_CONNECTION object. + +### Stream + +Represented by an QUIC_STREAM object. + +As per [RFC 9000 2.3 Stream Prioritization], streams should contain a priority +provided by the calling application. For MVP, this is not required to be +implemented because only one stream is supported. However, packets being +retransmitted should be preferentially sent as noted in +[RFC 9000 13.3 Retransmission of Information]. + +```c +void SSL_set_priority(SSL *stream, uint32_t priority); +uint32_t SSL_get_priority(SSL *stream); +``` + +For protocols where priority is not meaningful, the set function is a noop and +the get function returns a constant value. + +### Frame + +QUIC frames are represented by a leading variable length integer +indicating the type of the frame. This is followed by the frame data. +Only the first byte of the type is important because there are no defined +packet types that need more than one byte to represent. Thus: + +```c +struct ossl_quic_frame_st { + unsigned char type; +}; + +typedef struct ossl_quic_frame_st OSSL_QUIC_FRAME; + +struct ossl_quic_txp_frame_st { + OSSL_QUIC_FRAME *frame; /* Frame in wire format */ + size_t frame_len; /* Size of frame */ + uint32_t priority; /* Priority of frame */ +}; + +typedef struct ossl_quic_txp_frame_st OSSL_QUIC_TXP_FRAME; +``` + +The packetiser/ACK manager can alter the priority of a frame a small amount. +For example, a retransmitted frame may have it's priority increased slightly. + +#### Frames + +Frames are taken from [RFC 9000 12.4 Frames and Frame Types]. + +| Type | Name | I | H | 0 | 1 | N | C | P | F | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 0x00 | padding | ✓ | ✓ | ✓ | ✓ | ✓ | | ✓ +| 0x01 | ping | ✓ | ✓ | ✓ | ✓ | | | | | +| 0x02 | ack 0x02 | ✓ | ✓ | | ✓ | ✓ | ✓ | | | +| 0x03 | ack 0x03 | ✓ | ✓ | | ✓ | ✓ | ✓ | | | +| 0x04 | reset_stream | | | ✓ | ✓ | | | | | +| 0x05 | stop_sending | | | ✓ | ✓ | | | | | +| 0x06 | crypto | ✓ | ✓ | | ✓ | | | | | +| 0x07 | new_token | | | | ✓ | | | | | +| 0x08 | stream 0x08 | | | ✓ | ✓ | | | | ✓ | +| 0x09 | stream 0x09 | | | ✓ | ✓ | | | | ✓ | +| 0x0A | stream 0x0A | | | ✓ | ✓ | | | | ✓ | +| 0x0B | stream 0x0B | | | ✓ | ✓ | | | | ✓ | +| 0x0C | stream 0x0C | | | ✓ | ✓ | | | | ✓ | +| 0x0D | stream 0x0D | | | ✓ | ✓ | | | | ✓ | +| 0x0E | stream 0x0E | | | ✓ | ✓ | | | | ✓ | +| 0x0F | stream 0x0F | | | ✓ | ✓ | | | | ✓ | +| 0x10 | max_data | | | ✓ | ✓ | | | | | +| 0x11 | max_stream_data | | | ✓ | ✓ | | | | | +| 0x12 | max_streams 0x12 | | | ✓ | ✓ | | | | | +| 0x13 | max_streams 0x13 | | | ✓ | ✓ | | | | | +| 0x14 | data_blocked | | | ✓ | ✓ | | | | | +| 0x15 | stream_data_blocked | | | ✓ | ✓ | | | | | +| 0x16 | streams_blocked 0x16 | | | ✓ | ✓ | | | | | +| 0x17 | streams_blocked 0x17 | | | ✓ | ✓ | | | | | +| 0x18 | new_connection_id | | | ✓ | ✓ | | | ✓ | | +| 0x19 | retire_connection_id | | | ✓ | ✓ | | | | | +| 0x1A | path_challenge | | | ✓ | ✓ | | | ✓ | | +| 0x1B | path_response | | | | ✓ | | | ✓ | | +| 0x1C | connection_close 0x1C | ✓ | ✓ | ✓ | ✓ | ✓ +| 0x1D | connection_close 0x1D | | | ✓ | ✓ | ✓ | | | | +| 0x1E | handshake_done | | | | ✓ | | | | | + +The various fields are as defined in RFC 9000. + +##### Pkts + +_Pkts_ are defined as: + +| Pkts | Description| +| :---: | --- | +| I | Valid in Initial packets| +| H | Valid in Handshake packets| +| 0 | Valid in 0-RTT packets| +| 1 | Valid in 1-RTT packets| + +##### Spec + +_Spec_ is defined as: + +| Spec | Description | +| :---: | --- | +| N | Not ack-eliciting. | +| C | does not count toward bytes in flight for congestion control purposes. | +| P | Can be used to probe new network paths during connection migration. | +| F | The contents of frames with this marking are flow controlled. | + +For `C`, `N` and `P`, the entire packet must consist of only frames with the +marking for the packet to qualify for it. For example, a packet with an ACK +frame and a _stream_ frame would qualify for neither the `C` or `N` markings. + +### Packets + +Frames are coalesced into packets which are then sent by the record layer. +The `packet_header` is a pointer to the leading bytes of the packet. +The `frames` are pointers to the individual frames that make up the +packet's body. +It is expected that the record layer will encrypt from the `packet_header` and +`frames` directly without a copy. + +```c +enum packet_validity_e { + QUIC_PACKET_INITIAL, + QUIC_PACKET_HANDSHAKE, + QUIC_PACKET_0_RTT, + QUIC_PACKET_1_RTT +}; + +typedef enum packet_validity_e PACKET_VALIDITY; + +struct ossl_quic_packet_st { + QUIC_CONNECTION *conn; + unsigned char *packet_header; + size_t packet_header_length; + STACK_OF(OSSL_QUIC_TXP_FRAME) *frames; + + QUIC_PN packet_number; /* RFC 9000 12.3 */ + size_t packet_length; + + /* + * One of the QUIC_PN_SPACE_* values. This qualifies the pkt_num field + * into a packet number space. + */ + unsigned int pkt_space : 2; + + /* Pkts options */ + PACKET_VALIDITY validity; + + /* Spec */ + unsigned int no_ack : 1; + unsigned int no_congestion_control : 1; + unsigned int probing : 1; + unsigned int flow_controlled : 1; +}; + +typedef struct ossl_quic_packet_st OSSL_QUIC_PACKET; +``` + +#### Notes + +- Do we need the distinction between 0-rtt and 1-rtt when both are in + the Application Data number space? +- 0-RTT packets can morph into 1-RTT packets and this needs to be handled by + the packetiser. + +Interactions +------------ + +The packetiser needs to interact with other modules. This defines the APIs +by which it does so. + +Frames are passed to the packetiser on a per stream basis. +The frames must be fully formed. By passing a frame to this function, +ownership is passed to the packetiser which queues the frames for later +sending by the record layer. + +```c +int ossl_quic_packetiser_buffer_frame(OSSL_QUIC_TX_PACKETISER *tx, + QUIC_CONNECTION *stream, + const OSSL_QUIC_FRAME *frame, + size_t frame_length); +``` + +### Stream Send Buffers + +Data from the stream send buffers is treated specially. The packetiser knows +how much space is left in each packet and it will request that amount of data +from the stream send buffers. The stream send buffers will return a +constructed frame header and a pointer to the steam data and length. A second +call exists to allow the packetiser to know how much data is queued for a stream +so that planning for the creation of multiple packets is possible. + +```c +int ossl_quic_get_app_data(QUIC_STREAM *stream, size_t request, + const OSSL_QUIC_FRAME **frame, + const unsigned char **data, + size_t *data_len); + +size_t ossl_quic_get_app_data_size(QUIC_STREAM *stream); +``` + +#### Notes + +* Unclear how to best free the data after sent data was acked. + The data will be fragments from the buffers so the stream send buffers will + need to remember which fragment have been sent and which are pending and + only free once everything is sent: + +```c +int ossl_quic_free_app_data(QUIC_STREAM *stream, void *data, size_t data_len); +``` + +* Need a call to tell the stream send buffers to forget about previously + requested app data because it needs to be retransmitted and the + boundaries could change. Any record of the indicated data having being + transmitted should be removed and the data is made eligible to be sent + again. + +```c +int ossl_quic_retransmitting_app_data(QUIC_STREAM *stream, + void *data, size_t data_len); +``` + +### TLS Handshake Record Layer + +Uses the Record Layer API to implement the inner TLS-1.3 protocol handshake. +It produces the QUIC crypto frames which are queued using the same mechanism +as the [Stream Send Buffers](#stream-send-buffers) above. + +### Flow Controller and Statistics Collector + +To make decisions about what frames to coalesce, the packetiser relies +on the flow controller to enforce stream and connection bandwidth limits +[RFC 9000 4.1 Data Flow Control]. + +```c +/* + * Return the maximum amount of data that is permitted for the given stream. + * This includes both the stream limit and it's associated connection limit. + */ +size_t ossl_quic_stream_flow_maximum_size(QUIC_STREAM *stream); + +/* + * Inform the flow controller that an amount of data has been queued for + * sending to a stream. + */ +int ossl_quic_flow_controller_sent_data(QUIC_FLOW_CONTROLLER *flow, + QUIC_STREAM *stream, size_t bytes); +``` + +### Congestion Controller + +Also part of the frame coalescing decision is the congestion controller +[RFC 9002]. For MVP, this will be a _just send it_. + +```c +/* + * Pluggable congestion controller APIs go here + * Extract that is required from #18018 + */ +``` + +### QUIC Write Record Layer + +Coalesced frames are passed to the QUIC record layer for encryption and sending. +To send accumulated frames as packets to the QUIC Write Record Layer: + +```c +int ossl_qtx_write_pkt(OSSL_QTX *qtx, const OSSL_QTX_PKT *pkt); +``` + +The packetiser will attempt to maximise the number of bytes in a packet. +It will also attempt to create multiple packets to send simultaneously. + +The packetiser should also implement a wait time to allow more data to +accumulate before exhausting it's supply of data. The length of the wait +will depend on how much data is queue already and how much space remains in +the packet being filled. Once the wait is finished, the packets will be sent +by calling: + +```c +void ossl_qtx_flush_net(OSSL_QTX *qtx); +``` + +The write record layer is responsible for coalescing multiple QUIC packets +into datagrams. + +### ACK Handling and Loss Detector + +1. When a packet is sent, the packetiser needs to inform the ACK Manager. +2. When a packet is ACKed, inform packetiser so it can drop sent frames. +3. When a packet is lost, inform packetiser to create retransmission packet(s). +4. When a packet is discarded without ACK/loss, inform packetiser to clean up. + +```c +int ossl_ackm_on_tx_packet(OSSL_ACKM *ackm, OSSL_ACKM_TX_PKT *pkt) +int ossl_quic_packet_acked(OSSL_QUIC_TX_PACKETISER *tx, + OSSL_QUIC_PACKET *packet); +int ossl_quic_packet_lost(OSSL_QUIC_TX_PACKETISER *tx, + OSSL_QUIC_PACKET *packet); +int ossl_quic_packet_discarded(OSSL_QUIC_TX_PACKETISER *tx, + OSSL_QUIC_PACKET *packet); +``` + +#### Notes + +| Name here | Name in ACK Manager | +| --- | --- | +| `ossl_quic_packet_sent` | `QUIC_ACKM_on_tx_ack_packet` | +| `ossl_quic_packet_acked` | `on_acked` | +| `ossl_quic_packet_lost` | `on_lost` | +| `ossl_quic_packet_discarded` | `on_discarded` | + +Packets +------- + +Packets formats are defined in [RFC 9000 17.1 Packet Formats]. + +### Packet types + +QUIC supports a number of different packets. The combination of packets of +different types as per [RFC 9000 12.2 Coalescing Packets], is done by the +record layer. + +#### Version Negotiation Packet + +Refer to [RFC 9000 17.2.1 Version Negotiation Packet]. + +#### Initial Packet + +Refer to [RFC 9000 17.2.2 Initial Packet]. + +#### Handshake Packet + +Refer to [RFC 9000 17.2.4 Handshake Packet]. + +#### App Data 0-RTT Packet + +Refer to [RFC 9000 17.2.3 0-RTT]. + +#### App Data 1-RTT Packet + +Refer to [RFC 9000 17.3.1 1-RTT]. + +#### Retry Packet + +Refer to [RFC 9000 17.2.5 Retry Packet. + +Packetisation and Processing +---------------------------- + +### Application data frames + +The packetiser builds application data frames after requesting a specific +amount of application data. If insufficient data is available, or buffer +boundaries prevent fulfilling the entire request, the stream send buffer module +is free to return a smaller amount of data. + +### Retransmission + +When a packet is determined to be lost by the ACK Manager, the +`ossl_quic_packet_lost()` function will be called. This function will +extract the frame references from the packet and re-queue them for +transmission as if `ossl_quic_packetiser_buffer_frame()` had been called +for each +frame followed by `ossl_quic_packetiser_send_packets()`. Frames that need to be +retransmitted will be be considered higher priority than other pending +frames, although both types are available to construct packets from. +Moreover, any such constructed packets will not be subject to a delay +before transmission. + +### Restricting packet sizes + +Three factors impact the size of packets that can be sent: + +* MTU restricting packet sizes +* Flow control +* Congestion control + +The MTU limits the size of an individual packet, the other two limit the +total amount of data that can be sent. The packetiser needs to query the +current limits using the `ossl_quic_stream_flow_maximum_size()`, +`get_send_allowance()` and `get_data_mtu()` calls. + +The packetiser will prioritise sending [`C`](#spec) spec packets together +in order to maximise the amount of data available for the application. + +### Stateless Reset + +Refer to [RFC 9000 10.3 Stateless Reset]. It's entirely reasonable for +the state machine to send this directly and immediately if required. + +[RFC 9000 2.3 Stream Prioritization]: https://datatracker.ietf.org/doc/html/rfc9000#section-2.3 +[RFC 9000 4.1 Data Flow Control]: https://datatracker.ietf.org/doc/html/rfc9000#section-4.1 +[RFC 9000 10.3 Stateless Reset]: https://datatracker.ietf.org/doc/html/rfc9000#section-10.3 +[RFC 9000 12.2 Coalescing Packets]: https://datatracker.ietf.org/doc/html/rfc9000#section-12.2 +[RFC 9000 12.4 Frames and Frame Types]: https://datatracker.ietf.org/doc/html/rfc9000#section-12.4 +[RFC 9000 13.3 Retransmission of Information]: https://datatracker.ietf.org/doc/html/rfc9000#section-13.3 +[RFC 9000 17.1 Packet Formats]: https://datatracker.ietf.org/doc/html/rfc9000#section-17 +[RFC 9000 17.2.1 Version Negotiation Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.1 +[RFC 9000 17.2.2 Initial Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.2 +[RFC 9000 17.2.3 0-RTT]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.3 +[RFC 9000 17.2.4 Handshake Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.4 +[RFC 9000 17.2.5 Retry Packet]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.2.5 +[RFC 9000 17.3.1 1-RTT]: https://datatracker.ietf.org/doc/html/rfc9000#section-17.3.1 +[RFC 9002]: https://datatracker.ietf.org/doc/html/rfc9002 |