Hi, all
We are planning to implement a kernel module called COLO Proxy to buffer and
compare packets. This module is one of the important component of COLO project
and now it is still in early stage, so any comments and feedback are warmly
welcomed, thanks in advance.
=====
# RFC: COLO-Proxy Module
## Rationale
COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
is conducted.
Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO
By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new kernel module which called
colo-proxy.
This document describes the design of the colo-proxy module
## Glossary
* PVM - Primary VM, which provides services to clients.
* SVM - Secondary VM, a hot standby and replication of PVM.
* PN - Primary Node, the host which PVM runs on
* SN - Secondary Node, the host which SVM runs on
## Network topology
================================= Normal =====================================
+--------+
| client |
+----+---+
-------------------------+ | + -------------------------+
PN | + | SN|
+-------+ +----[eth0]-----[switch]-----[eth0]---------+ |
|PVM | +---+-+ | | +---+-+ |
| [tap0]--+ br0 | | | | br0 | |
| | +-----+ [eth1]-----[forward]----[eth1]--+ +-----+ |
+-------+ | | | +-------+|
| | | +-----+ | SVM||
[eth2]---[checkpoint]---[eth2] +--+ br1 |-[tap0] ||
| | +-----+ | ||
| | +-------+|
-------------------------+ +--------------------------+
e.g.
PN:
br0: 192.168.0.33
eth1: 192.168.1.33
eth2: 192.168.2.33
SN:
br0: 192.168.0.88
br1: no ip address
eth1: 192.168.1.88
eth2: 192.168.2.88
============================== After failover ================================
+--------+
| client |
+----+---+
-------------------------+ | ---------------------------+
PN (dead) | + | SN (alive)|
+-------+ +----[eth0]--X--[switch]-----[eth0]-------+ |
|PVM | +---+-+ | | +---+-+ |
| [tap0]--+ br0 | | | | br0 +--+ |
| | +-----+ [eth1]--X--[forward]----[eth1] +-----+ | |
+-------+ | | | +-------+|
| | +-----+ | | SVM||
[eth2]-X-[checkpoint]---[eth2] | br1 | +[tap0] ||
| | +-----+ | ||
| | +-------+|
-------------------------+ +--------------------------+
## Network flow
### Receive packets from client (Input)
+------+
|Client|
+---+--+
+-----------------------+ | +------------------------+
|PN | v | SN|
| +---[eth0]<---[switch] | +--------+ |
| +-------+ v | | | SVM | |
| | PVM | +-+-+ | | [tap0] | |
| | [tap0]<-+br0| | | ^ | | |
| | | | +---+ | | | +--------+ |
| +-------+ | | | +-+-------------+ |
| +-------->[eth1]------------->[eth1]--->colo-proxy | |
| copy&forward| | |*Adjust | |
| | | | Client's ack | |
+-----------------------+ +-----+---------------+--+
* colo-proxy on SN:
** Capture the first ack from client, find out the initial seq number of the
tcp connection on PVM. (for seq number adjustment)
** Adjust ack/sack from client until next checkpoint, make sure tcp
connection on SVM won't break.
### Response packets (Output)
+------+
|Client|
+---^--+
+----------------------------+ | +------------------------+
|PN + + | SN|
| +----+ checkpoint +-->[eth0]+-->[switch] | +---------+ |
| |PVM | ^ | | + | + SVM | |
| +-+--+ | v +-+-+ | | [tap0] | |
| | |[tap0]->br0| | | + + | |
+---v--+ | ^ +---+ + + | +---------+ |
||Vhost| | | ++[eth1]<------------+[eth1]<---+v-------------+ |
+---+--+ | | | + + |colo-proxy | |
| | No | |Yes | | | |*Adjust SVM's | |
+---|--------|--|--------|---+ | | Seq number | |
| | identical? | | +------+--------------+--+
| +-v-----+ ^ +-----v-+ |
| |enqueue+---+ |enqueue| |
| +-------+compare +-------+ |
| |
| colo-proxy |
+----------------------------+
* colo-proxy on SN:
** track the initial seq number of the tcp connection on SVM.
(for seq number adjustment)
** Adjust seq number from SVM until next checkpoint.
* colo-proxy on PN:
** enqueue the packets from SVM
** enqueue the packets from PVM
** compare the tcp payload data of these two queue
** if the data is identical, release PVM queue, drop SVM queue
** if the data is not identical, notice the upper layer(userspace tools:
QEMU or libxl on Xen) a checkpoint is needed
** release PVM queue and drop SVM queue at checkpoint
### After failover
At this point, PN is dead, SVM is serving the clients.
#### Receive packets from client (Input)
+------+
|Client|
+---+--+
|
+---v--+
|Switch|
+---+--+
v
+-------------[eth0]--------------+
| |-------+ SN |
| +------v---------+ |
| |colo-proxy | |
| |*Adjust client's| |
| | ack number | |
| +------+---------+ |
| | |
| | +-----------+ |
| | | SVM | |
| +--->[tap0] | |
| | | |
| +-----------+ |
+---------------------------------+
* colo-proxy on SN:
** Adjust the ack/sack number from client, this only applies to the existing
tcp connection.
#### Response packets (Output)
+------+
|Client|
+---^--+
|
+---+--+
|Switch|
+---^--+
+
+-------------[eth0]--------------+
| |-------^ SN |
| +----------------+ |
| |colo-proxy | |
| |*Adjust SVM's | |
| | seq number | |
| +------^---------+ |
| | |
| | +-----------+ |
| | | SVM | |
| +---+[tap0] | |
| | | |
| +-----------+ |
+---------------------------------+
* colo-proxy on SN:
** Adjust the seq number of the packets returned by SVM, this only applies
to the existing tcp connection.
NOTE:
We track the initial seq number of the tcp connection on both PVM/SVM so that
we can calculate the offset when we do the seq adjustment after failover.
## Implementation
We archive our goal by extending nf_conntrack mechanism.
There're 4 kernel modules in colo-proxy:
### nf_conntrack_colo
In this module We add an nf_conntrack extension named 'colo':
<pre>
static struct nf_ct_ext_type nf_ct_colo_extend __read_mostly = {
.len = sizeof(struct nf_conn_colo),
.move = nf_ct_colo_extend_move,
.destroy = nf_ct_colo_extend_destroy,
.align = __alignof__(struct nf_conn_colo),
.id = NF_CT_EXT_COLO,
};
</pre>
This extension hold essential states needed by colo-proxy. e.g. manage the
node status, the tcp connection status.
### xt_PMYCOLO
This module is for PN. It do the following operations:
* Register a xt_target(cooperate with iptables) to initiate the PN node
status, run a kernel thread to compare packets.
<pre>
static struct xt_target colo_primary_tg_regs[] __read_mostly = {
{
.name = "PMYCOLO",
.family = NFPROTO_UNSPEC,
.target = colo_primary_tg,
.checkentry = colo_primary_tg_check,
.destroy = colo_primary_tg_destroy,
.targetsize = sizeof(struct xt_colo_primary_info),
.table = "mangle",
.hooks = (1 << NF_INET_PRE_ROUTING),
.me = THIS_MODULE,
},
};
static int colo_primary_tg_check(const struct xt_tgchk_param *par)
{
/*
* Setup forward device, init primary node status, create kthread for
* packets comparison.
*/
}
</pre>
* Register a nf_queue_handler to enqueue packets sent by PVM.
<pre>
static const struct nf_queue_handler coloqh = {
.outfn = &colo_enqueue_packet,
};
</pre>
* Register some nf hooks to enqueue packets sent by SVM.
<pre>
static struct nf_hook_ops colo_primary_ops[] __read_mostly = {
{
.hook = colo_slaver_queue_hook,
.owner = THIS_MODULE,
.pf = NFPROTO_IPV4,
.hooknum = NF_INET_PRE_ROUTING,
.priority = NF_IP_PRI_RAW + 1,
},
{
.hook = colo_slaver_queue_hook,
.owner = THIS_MODULE,
.pf = NFPROTO_IPV6,
.hooknum = NF_INET_PRE_ROUTING,
.priority = NF_IP_PRI_RAW + 1,
},
{
.hook = colo_slaver_arp_hook,
.owner = THIS_MODULE,
.pf = NFPROTO_ARP,
.hooknum = NF_ARP_IN,
.priority = NF_IP_PRI_FILTER + 1,
},
};
</pre>
### xt_SECCOLO
This module is for SN. It do the following operations:
* Register a xt_target(cooperate with iptables) to initiate the SN node
status.
<pre>
static struct xt_target colo_secondary_tg_regs[] __read_mostly = {
{
.name = "SECCOLO",
.family = NFPROTO_UNSPEC,
.target = colo_secondary_tg,
.checkentry = colo_secondary_tg_check,
.destroy = colo_secondary_tg_destroy,
.targetsize = sizeof(struct xt_colo_secondary_info),
.table = "mangle",
.hooks = (1 << NF_INET_PRE_ROUTING),
.me = THIS_MODULE,
},
};
</pre>
* Register some nf hooks to track the initial seq number of the tcp
connections on both PVM/SVM, and do the seq adjustment for SVM(by
using the existing nf_conntrack_seqadj module).
<pre>
static struct nf_hook_ops colo_secondary_ops[] __read_mostly = {
{
.hook = colo_secondary_hook,
.owner = THIS_MODULE,
.pf = NFPROTO_IPV4,
.hooknum = NF_INET_PRE_ROUTING,
.priority = NF_IP_PRI_MANGLE + 1,
},
{
.hook = colo_secondary_hook,
.owner = THIS_MODULE,
.pf = NFPROTO_IPV6,
.hooknum = NF_INET_PRE_ROUTING,
.priority = NF_IP_PRI_MANGLE + 1,
},
};
</pre>
### nfnetlink_colo
This module is for communication with the userspace tools like QEMU or libxl.
In this module, add a colo protocol to the existing nfnetlink mechanism.
<pre>
static const struct nfnetlink_subsystem nfulnl_subsys = {
.name = "colo",
.subsys_id = NFNL_SUBSYS_COLO,
.cb_count = NFCOLO_MSG_MAX,
.cb = nfnl_colo_cb,
};
static const struct nfnl_callback nfnl_colo_cb[NFCOLO_MSG_MAX] = {
[NFCOLO_KERNEL_NOTIFY] = { .call = NULL,
.policy = NULL,
.attr_count = 0, },
[NFCOLO_DO_CHECKPOINT] = { .call = colo_do_checkpoint,
.policy = nfnl_colo_policy,
.attr_count = NFNL_COLO_MAX, },
[NFCOLO_DO_FAILOVER] = { .call = colo_do_failover,
.policy = nfnl_colo_policy,
.attr_count = NFNL_COLO_MAX, },
[NFCOLO_PROXY_INIT] = { .call = colo_init_proxy,
.policy = nfnl_colo_policy,
.attr_count = NFNL_COLO_MAX, },
[NFCOLO_PROXY_RESET] = { .call = colo_reset_proxy,
.policy = nfnl_colo_policy,
.attr_count = NFNL_COLO_MAX,},
};
</pre>
On 30.06, Li Zhijian wrote:
> |ping...
>
> and i have another question:
> can i add a new |||nf_ct_ext_id simply without touching the exiting kernel
> code?|
No, the kernel needs to know the highest extension ID in order to
allocate space for the offsets.
> in order to support COLO-Proxy, i need a extra nf_ct_ext_id called
> "NF_CT_EXT_COLO"
> to store some message of COLO related. This message can help COLO-Proxy to
> buffer packet and compare packet for each connection.
>
> Thanks
> Li Zhijian
Cheers,
Patrick
On 06/30/2015 05:19 PM, Patrick McHardy wrote:
> On 30.06, Li Zhijian wrote:
>> |ping...
>>
>> and i have another question:
>> can i add a new |||nf_ct_ext_id simply without touching the exiting kernel
>> code?|
>
> No, the kernel needs to know the highest extension ID in order to
> allocate space for the offsets.
So if we want to add a new extension, we should post the module too. Is
it right?
Thanks
Wen Congyang
>
>> in order to support COLO-Proxy, i need a extra nf_ct_ext_id called
>> "NF_CT_EXT_COLO"
>> to store some message of COLO related. This message can help COLO-Proxy to
>> buffer packet and compare packet for each connection.
>>
>> Thanks
>> Li Zhijian
>
> Cheers,
> Patrick
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> .
>