Contact Support

Customers who viewed this article also viewed

banner icon

Identify Changes in NetScaler build files with

File Integrity Monitoring

Learn More Watch Video
CTX226652 {{tooltipText}}

Basic Design Guidelines and Principles on NetScaler Routing, Default Routes, Interfaces and Channels, VLANs, and GARP

Applicable Products

  • Citrix ADC

Information

Routing and Default Routes

User-added image
 
1 - Interface 0/x on MPX or SDX Should Not be Used for Production Traffic. On an MPX or SDX, Interfaces named 0/x are referred to the Management Interfaces. This does not mean you must use these interfaces for Management. What it means is that these interfaces are NOT designed for Production traffic. They do not have the hardware buffers and optimization necessary in order to achieve sustained 1Gbps throughput. Therefore, If your Default Route is in the same Subnet as your NSIP, you need to either change the default route or use a 1/x interface for your Management network as the 1/x interfaces are fully optimized for production 1Gbps traffic. See https://support.citrix.com/article/CTX136926 for more information. NOTE This does not apply to a VPX.
  1. Option 1 – Don’t connect to Interfaces 0/x – Disconnect the cable from interface 0/1. NetScaler will listen for the NSIP on the other interfaces. (NOTE: This is not an option for SDX, as the SVM and XenServer can only speak to 0/x interfaces)
  2. Option 2change the default route to a different interface as detailed in Section 2.
2 - Default gateway (route 0.0.0.0) should be on a Production network, and not on any 0/x interface. This applies to MPX or SDX instances, not a self-hosted VPX. When first setting up a NetScaler, it asks you for the NSIP, Subnet Mask, and Gateway address. The problem this creates for Administrators is that they essentially just configured their default route to be on their Management Network using Interface 0/1.
  1. To check what your routes are, run in CLI "show route" and your default gateway is the IP in the line where the Network and Netmask are 0.0.0.0. Here's an example where the Gateway is on Line 1:
    > sh route
            Network          Netmask          Gateway/OwnedIP  State   Traffic Domain  Type
            -------          -------          ---------------  -----   --------------  ----
    1)      0.0.0.0          0.0.0.0          10.25.213.65     UP      0              STATIC
    2)      127.0.0.0        255.0.0.0        127.0.0.1        UP      0              PERMANENT
    3)      10.25.213.64     255.255.255.192  10.25.213.68     UP      0              DIRECT
    4)      172.16.0.0       255.255.255.0    172.16.0.1       UP      0              DIRECT
  2. To check the Interface and VLAN used for your Default Gateway, check the ARP table using "sh arp" in CLI. You can also search for the specific IP using "show arp | grep 10.25.213.65". Here's an example where you see the Gateway 10.25.213.65 is using Interface 1/1 and VLAN 1:
    > sh arp
            IP               MAC                Iface VLAN  Origin     TTL     Traffic Domain
            --               ---                ----- ----  ------     ---     --------------
    1)      127.0.0.1        02:00:18:a4:00:1e  LO/1  1     PERMANENT  N/A    0
    2)      10.25.213.70     02:00:0f:46:00:28  1/1   1     DYNAMIC    967    0
    3)      10.25.213.68     02:00:18:a4:00:1e  LO/1  1     PERMANENT  N/A    0
    4)      10.25.213.67     02:00:0f:46:00:28  1/1   1     DYNAMIC    641    0
    5)      10.25.213.65     00:08:e3:ff:fd:90  1/1   1     DYNAMIC    483    0
  3. Change the default route to use a Gateway on your Production Subnet and Interface. Assume your Management network is 10.0.0.0/24 with gateway 10.0.0.1, and Production network is 10.1.1.0/24 with gateway 10.1.1.1. Setup your config like this:
    • SNIP: (Management Access Disabled) 10.1.1.2
    • NSIP: (Management Access Enabled) 10.0.0.2
    • Default Route: 0.0.0.0 0.0.0.0 10.1.1.1 (System > Network > Routes). This uses a router on the SNIP network instead of the NSIP network.
    • NOTE: changing the default gateway might break management traffic unless you configure static routes, a Policy Based Route or enable MAC Based Forwarding.
3 - Avoid Multiple Default Routes. Some environments may have multiple core stacks that can be used as a default route and the NetScaler is directly connected to both networks. In this case, you can configure multiple Default Routes in the NetScaler. However, care needs to be taken to ensure the additional default routes do not have the same cost or weight. If they do, you may experience various loss of connections as both routes will be actively used. To check this, you need to run this command: "sh route -detail" so you can see the Distance and cost. Example:
> sh route -detail
        Network          Netmask          Gateway/OwnedIP  State   Traffic Domain  Type
        -------          -------          ---------------  -----   --------------  ----
1)      0.0.0.0          0.0.0.0          10.25.213.65     UP      0              STATIC
        Distance:     1 Cost:     0     Weight:     1
1)      0.0.0.0          0.0.0.0          10.25.214.1     UP      0              STATIC
        Distance:     1 Cost:     50     Weight:     50

Interfaces, Channels, and VLANs

User-added image
  1. Don’t connect multiple interfaces/channels to the same VLAN, including VLAN 1:
    1. If you do not properly configure your VLANs, it can cause some unexpected packet routing in your network as well as Layer 2 looping any time there are more than one Active interface with the same VLAN (either Native or Tagged).
    2. For Azure and other Cloud environments: If multiple interfaces are in the same Vnet, you must use "dummy" native VLANs on the ADC and bind the SNIPs to those VLANs and the VLANs to the interfaces (as Native, Not Tagged) to prevent Interfaces from being temporarily disabled (aka Muted) and prevent errant failovers due to MAC Moves. Additionally, ensure that the in Azure, the SNIP bound to the NIC matches the Interface VLAN and SNIP bindings in the ADC, otherwise you will get Incomplete ARP in ADC. These steps are covered in Steps 2 and 3 below.
    3. By Default, all Interfaces and Channels are on Native VLAN 1. This creates 2 possible problems:
      • The NetScaler thinks all traffic received is on the same network, so it will use any interface to send the traffic out on; if you have a different Native VLAN on the interface it sent data out on, then the traffic will not be routed as you expect.
      • If the NetScaler receives broadcast packets on one port, it may retransmit on another port. If both switchports are on the same VLAN, you just created a Layer 2 loop.
    4. To remove an interface/channel from VLAN 1:
      • If you are not using Native VLANs on your switch interface/port channel: Change the Native VLAN on the NetScaler Interface/Channel to an unused VLAN number such as 999. Note that you should not use the same unused VLAN number for multiple Channels or Interfaces, as it will create a layer 2 loop.
      • If you are using Native VLANs on your switch interface/port channel: Change the Native VLAN on the NetScaler Interface/Channel to match. However take care not to have multiple active Interfaces or Channels on the same VLAN as doing so creates Layer 2 loops.
      • You cannot remove the Native VLAN. Instead, you can change it or set TagAll for the interface or channel.
        1. If the switch port isn’t configured with an untagged native VLAN, then enable tagall on the interface so High Availability heartbeat packets will be tagged. See this article for more information: https://support.citrix.com/article/CTX115575
    5. To view the Native VLAN on an interface, run "sh interface" in CLI. This will also inform you if the interface is using the TAGALL option.
  2. Bind an Interface to your VLAN - The NetScaler, by default, does not attach a new VLAN to an interface. This means the VLAN is not going to be used until you bind it to an interface. When the new VLAN is not bound to an Interface and that VLAN is Tagged, the NetScaler will drop all inbound traffic from that VLAN.  Also, do not bind the same VLAN to more than one interface.
  3. Bind Subnets to your VLANs - The NetScaler does not work like a typical Router. Most Routers attach IPs to Interfaces. On a NetScaler, the IPs float on any Interface unless configured otherwise. Therefore, any Subnet you want to ensure the NetScaler will send over a specific VLAN, especially when the NetScaler is Initiating that traffic, then you need to bind a SNIP within that subnet to the VLAN.
    1. A common argument we hear against this is that it used to work fine and now it doesn't without binding the Subnet to the VLAN. This often occurs because the NetScaler will learn which VLAN to send traffic out, but this can take time as it builds it's ARP tables. After a reboot or firmware upgrade, as it starts building the ARP tables again, it may initially learn and therefore be using a different path than you desire, such as your default route. It's best to instruct it which path to take by Binding the SNIP to the VLAN. Once a SNIP is bound to a VLAN, the entire subnet for that SNIP will be bound to the VLAN.
    2. Ensure every SNIP is bound to a VLAN (except in cases where you have more than 1 SNIP in a subnet, then you only need to bind one), and that the VLAN, in turn, is bound to only one Interface or channel. It is also often best to have a SNIP in every Subnet, but that is not required as the most specific Route will be used for any destination Subnet that does not have a SNIP.
  4. To identify the VLAN and Interface Used by a Subnet:
    1. Go to System->Network->VLANs.
    2. Edit each VLAN configured, in turn, until you find the correct IP as explained in the next Step.
    3. Click the IP Bindings tab to see which IP, and thus which Subnet, is bound and thus is using this VLAN.
    4. Once you identify the VLAN that has an IP bound to it, where that IP is within the subnet of the Default Route, then click the Interface Bindings.
    5. Each Interface or Channel bound to this VLAN will be used.
Example:
Assume the Default route is "0.0.0.0 0.0.0.0 10.1.1.1".
Assume you have two SNIPs of 10.0.0.5 and 10.1.1.69. Since 10.1.1.69 is in the subnet of the default route, that is the one you want to look for. In the below screenshots, we are reviewing VLAN 1 and we see the IP 10.1.1.69 is bound to this VLAN, so we know we are looking at the correct VLAN.
Now Click Interface Bindings. In the VLAN Interface bindings we see that Interface 1/1 is used for this subnet, and therefore is used for the default route.

NOTE: If you do not have any IPs bound to your VLANs, then by default it will be sent out VLAN 1, so in that case just look at which Interfaces are bound to VLAN 1. This also means the NetScaler will not use your Configured VLANs for traffic it Initiates unless you bind an IP to the new VLAN.
User-added image  User-added image

GARP

If GARP does not work, use VMAC - By default, the NetScaler uses GARP to advertise it's IP to MAC address bindings to other network devices. This typically works without issue, however, as you create more services in the NetScaler, you may begin to experience issues when failing over on an HA pair. The most common issue is that services will remain down in the NetScaler you failed over to due to some network devices not having updated their ARP tables with the new MAC address. You can easily verify this by checking their ARP tables to see if the MAC addresses match those on the now-Primary NetScaler. When this occurs, it is highly likely that some of your network devices are limiting the number of GARP advertisements they honor. In this case it is necessary to configure VMAC on all your active interfaces and/or channels. If you expect to have a large configuration on your NetScaler, it may be best to configure VMAC for all interfaces and channels during the initial deployment.
  1. NOTE: Do not forget to configure VMAC for the Interface or Channel used by your Default Route.

Explanation of NetScaler IP Types

User-added image

NSIP - This is the NetScaler's IP. Generally the IP used for Management because it is the only IP unique to an individual NetScaler in an HA or Cluster environment. Also important to note is that LDAP, RADIUS, and User scripted Monitor traffic (such as the LDAP monitor and Storefront monitor) will Source from the NSIP and thus route over the VLAN and Interface the NSIP is bound to (Default Native VLAN 1). If you need the LDAP and RADIUS traffic to source from the SNIP, then create a LB Vserver for your backend servers.

SNIP - This is the NetScaler's Subnet IP. This IP is used to initiate communication to backend servers and is generally always going to initiate traffic. That said, it can be the destination for traffic in these cases:

  • It can be used as the Gateway address on other devices when doing Layer 3 routing on the NetScaler.
  • It can, when enabled, accept Management services, such as access to the GUI, SSH, and SNMP.

VIP - The VIP is unique in that it will never be used to initiate outbound traffic. It is intended to receive Traffic only. Of course once it receives traffic, it will reply and send traffic outbound back to the client. The point is, however, that the VIP does not and will not initiate the outbound traffic. Note this also means it will not be used as the source for communicating with Backend servers used in, for example, an LB Vserver.

MIP - Very similar to the SNIP. We won't go into details here as it's rarely used any more, but if you like you can review these links:

Viewing Specific IP Types in CLI. The standard CLI command for showing all IPs is "sh ip". However if you have a lot of Vservers configured, this will be a long list. If you only want to see your SNIPs or MIPs, you can use the -type switch. For example: "sh ip -type SNIP".

Additional Resources