UPGW对WIFI、5G共部署的支持

需求分析

逻辑拓扑

上图作为WIFI、5G共部署的逻辑拓扑，对UPGW提出了一些新的需求：

  1. nr0和nr1可以通过配置文件指定egress口，如nr0的流量透传至upf0，nr1的流量透传至upf1，反向亦然；

  2. 通过设置route规则可将nr0和nr1的流量卸载到dn0和dn1；

  3. dn0和dn1传入的流量可以通过查询learning表项，将流量转入upstream口或者downstream口。

现有代码支持度

  1. 上述第一项需求完全支持，不需要重新开发；

  2. 上述第二项不支持，只能将命中route规则的流量转发至第一个lbp口，比如nr0和nr1的流量都送到了dn0；

  3. 上述第三项部分支持，非arp报文可以实现上面功能，若是arp报文，因为UPGW实现了arp代答机制，会将回应的arp包传入最后配置的一个lbp口，所以前面配置的lbp口都收不到arp reply，无法进行后续报文发送。

开发项1

  • 目标

    nr端口的流量可以指定lbp端口转发至对应的dn。

  • 解决思路

    通过命令行下发route规则时，再指定对应的lbp端口，使命中此规则的流量走指定的ring转发报文。

  • 代码实现

    1. 在命令行下发端不做更改，相当于在nes_cmdline_acl_string_t route_data中加入了一组新的键值：

      typedef struct cmd_ctrl_route_add_result {
    cmdline_fixed_string_t add_string;
    cmdline_fixed_string_t route_string;
    struct ether_addr mac_addr;
    char route_rule_uuid[37];
    nes_cmdline_acl_string_t route_data; /* 存放下发的键值对 */
} cmd_ctrl_route_add_result;

    2. 在解析端nts_acl_cfg_lookup_prepare函数中增加对该键值对的解析：

      else if (strcmp("dst_lbp_port", field_name) == 0) {
    ret = FIELD_ROUTE_DST_LBP_PORT; /* 匹配name得到对应的枚举 */

      case FIELD_ROUTE_DST_LBP_PORT:
{
    char *value = cfg_lookup_entries[i].value;
    int j = 0;
    char port_name[PORT_NAME_SIZE];
    const char *buffer;

    if (strlen(value) > NTS_ACL_CFG_MAX_LBP_VALUE_LEN)
    {
        NES_LOG(ERR, "Value %s is too long\n", value);
        return NES_FAIL;
    }
    for (j = 0; j < strlen(value); j++) {
        if (0 == isdigit(value[j])) {
            NES_LOG(ERR, "Value %s is not legitimate\n", value);
            return NES_FAIL;
        }
    }

    *dst_lbp_port  = atoi(value); /* 根据枚举去解析value得到传入的port */
    snprintf(port_name, PORT_NAME_SIZE, PORT_SECTION_NAME"%d", *dst_lbp_port);
    if (NES_SUCCESS != nes_cfgfile_entry(port_name, TRAFFIC_DIRECTION, &buffer)) {
        NES_LOG(ERR, "Unable to find "TRAFFIC_DIRECTION" entry for port %d\n", *dst_lbp_port);
        return NES_FAIL;
    }
    if (0 != strncmp(buffer, TRAFFIC_DIRECTION_LBP,sizeof(TRAFFIC_DIRECTION_LBP)))     {
        NES_LOG(ERR, "Port %d is not "TRAFFIC_DIRECTION_LBP"\n", *dst_lbp_port);
        return NES_FAIL;
    }

    break;
}

    3. 在nts_acl_lookup_add_impl函数中将解析到的dst_lbp_id转化为ring，写入对应的entry：

      if (dst_lbp_port >= 0)
{
    snprintf(tx_ring_name, sizeof(tx_ring_name),
             PORT_TX_QUEUE_NAME_TEMPLATE, dst_lbp_port);
    ring_name = tx_ring_name;
}
else
{
    NES_LOG(ERR, "Can not find dst_lbp_port\n");
    return NES_FAIL;
}

开发项2

  • 目标

    dn发出的arp请求报文进入UPGW后，arp代答报文还能从源端口回来。

  • 解决思路

    arp报文命中learning规则后，将arp代答的报文发送给源端口。

  • 代码实现

    1. 定义dpdk端口和逻辑端口互相转换的数组，后期为了方便扩展，会将数组变为为hash table：

      char pdk_port_to_logic_port[MAX_DPDK_PORT_NUM];

    2. 在nes_dev_port_new_device函数中获取dpdk端口和配置文件中逻辑端口的映射关系：

      if (NES_SUCCESS != nes_dev_eth_find_port_id_by_pci(&pci_addr_cfg, &dpdk_port_id)){
    NES_LOG(ERR, "Unable to find port id for pci_addr=%s\n", buffer);
    break;
}

if (dpdk_port_id > MAX_DPDK_PORT_NUM - 1) {
    NES_LOG(ERR, "dpdk_port_id %d is too large\n", dpdk_port_id);
    break;
}
dpdk_port_to_logic_port[dpdk_port_id] = i;

    3. 在nts_edge_arp_edit函数中，通过mbuf中存放的port获取对应的逻辑端口，做到去源端口回包：

      if (dpdk_port_to_logic_port[mbuf->port] == DPDK_PORT_INIT)
{
    NES_LOG(ERR, "Can not find logic port\n");
    return NES_FAIL;
}
snprintf(buffer, sizeof(buffer), DOMAIN_PORT_TX_QUEUE_NAME_TEMPLATE, dpdk_port_to_logic_port[mbuf->port]);

测试验证

  • 配置文件

    [PORT0]
name = 0000:00:04.0
description = 82540EM Gigabit Ethernet Controller
pci-address = 0000:00:04.0
traffic-type = IP
traffic-direction = upstream
egress-port = 1

[PORT1]
name = 0000:00:05.0
description = 82540EM Gigabit Ethernet Controller
pci-address = 0000:00:05.0
traffic-type = IP
traffic-direction = downstream
egress-port = 0

[PORT2]
name = 0000:00:06.0
description = 82540EM Gigabit Ethernet Controller
pci-address = 0000:00:06.0
traffic-type = IP
traffic-direction = lbp
egress-port = 0
lbp-mac = fa:16:3e:c0:70:fa

[PORT3]
name = 0000:00:0e.0
description = 82540EM Gigabit Ethernet Controller
pci-address = 0000:00:0e.0
traffic-type = IP
traffic-direction = upstream
egress-port = 4

[PORT4]
name = 0000:00:0f.0
description = 82540EM Gigabit Ethernet Controller
pci-address = 0000:00:0f.0
traffic-type = IP
traffic-direction = downstream
egress-port = 3

[PORT5]
name = 0000:00:10.0
description = 82540EM Gigabit Ethernet Controller
pci-address = 0000:00:10.0
traffic-type = IP
traffic-direction = lbp
egress-port = 3
lbp-mac = fa:16:3e:a7:02:cc

[VM common]
max = 32
number = 2
vhost-dev = /var/lib/appliance/nts/qemu/usvhost-1

[NES_SERVER]
;ctrl_socket = /var/lib/appliance/nts/control-socket
;ctrl_ip = 127.0.0.1
ctrl_ip = 0.0.0.0
ctrl_port = 8080

[KNI]
max = 32

[REDIS_SERVER]
host = 127.0.0.1
port = 6379

  • route规则

    route add fa:16:3e:c0:70:fa ace4d210-3d92-422b-83d7-11a0de50fea5  prio:98,srv_ip:192.168.30.116/32,encap_proto:noencap,dst_lbp_port:2

    route add fa:16:3e:a7:02:cc ace4d210-3d92-422b-83d7-11a0de50fea6 prio:98,srv_ip:192.168.60.22/32,encap_proto:noencap,dst_lbp_port:5

  • 功能测试1

    UPGW启动后先通流量，进行lerrning表项学习，之后下发上面的route规则；nr0到upf0的ping包被卸载到了dn0，目的mac被改为了dn0的mac；nr1到upf1的ping包被卸载到了dn1，目的mac被改为了dn1的mac。下图为两个dn收到的卸载流量：

    image-20210608133554342

    image-20210608134249906

  • 功能测试2

    dn0向nr0测试ping包，代答arp由UPGW发给dn0，之后向dn0发送icmp request报文，目的mac为nr0；dn1向nr1测试ping包，代答arp由UPGW发给dn1，之后向dn1发送icmp request报文，目的mac为nr1。下图是收到的arp reply和之后进行的回包：

    image-20210608104944151

    image-20210608105230136