Wednesday, August 17, 2011

Juniper Aggregate Interfaces (LACP/No LACP)

Last week, I was implementing link aggregation on Juniper EX Series switches (EX4200). I was moving them off HP switches and putting them into Juniper.
We were moving VMWare ESX boxes, NetApp and some physical boxes, though the configuration is very simple, I would like to mention how to do those with these boxes.
Points to remember:
  • You will explicitly need to create “Aggregated Ethernet” interfaces on juniper
  • You cannot have more than 64 AE interfaces on a single Virtual Chassis
  • You cannot change the algorithm for load balancing between the multiple links. (If you are interested in the Junipers LAG algorithm read this )
Now having said this, lets look at the common steps, you need to create the Aggregated Ethernet interfaces on the Juniper.
root@EX4200#set chassis aggregated-devices ethernet device-count 4
The above command will create 4 “ae” interfaces, after you commit, you should see ae0 – ae 3 created. You can have any value between 1 and 64 in the place of the 4.
Please also remember that, the individual interfaces should have absolutely no configuration before they can be added to the aggregation. In our example, we have taken interfaces ge-0/0/0 and ge-0/0/1
root@EX4200#delete interfaces ge-0/0/0 unit 0
root@EX4200#delete interfaces ge-0/0/1 unit 0
You may chose to completely delete the interface itself, so all the configuration under the interfaces (like mtu) will be cleared.
At this point, you will have some thing similar
chassis {
aggregated-devices {
ethernet {
device-count 4;
ge-0/0/0 {
ge-0/0/1 {

So, now that we have created the place holders for the aggregated interfaces, we will look into LACP in a little bit more detail.

Please remember that LACP is a negotiation protocol. It’s a standard protocol comparable to Cisco PAgP, which was later standardized into LACP. Also, remember that both ends must support LACP for the bundling to work using LACP.

ESX and juniper EX:

When creating the Aggregated interface, remember that the ESX (version 4) doesn’t support LACP, so we will be doing a generic aggregation. In this example, the ESX ports are vmnic2 and vmnic3 and the Switch ports are ge-0/0/0 and ge-0/0/1

  • Add the multiple networks in a vSwitch
  • Set Load Balancing as “Route based on IP Hash”
  • Network Failover Detection as “Link Status Only”
  • Notify Switch as Yes


That’s all the change you need on VMWare.

 On Juniper EX

  • Connect vmnic2 => ge-0/0/0
  • Connect vmnic3 => ge-0/0/1

1: set interface ge-0/0/0 ether-options 802.3ad ae0
2: set interface ge-0/0/1 ether-options 802.3ad ae0
3: set ae5 unit 0 description "ESX - LAN Bundle"
4: set ae5 unit 0 family ethernet-switching port-mode access vlan members vlan1

So, with this, you see, we havent set any LACP properties, this is the way ESX likes is Smile. We should have the interfaces up.

You can make sure by

1: root@EX4200> show interfaces terse | match ae0
2: ge-0/0/0.0 up up aenet --> ae0.0
3: ge-0/0/1.0 up up aenet --> ae0.0
4: ae0 up up
5: ae0.0 up up eth-switch

If the ae interface status is down, which means some thing went wrong and you need to reconfigure.

There is one thing I forgot to mention, some times you will lose connectivity while doing the change. In that case, you would have configured the Management Network differently, but the source Hash will not be working...

Please use the following to correct it ...

"# vim-cmd hostsvc/net/vswitch_setpolicy --nicteaming-policy=loadbalance_ip vSwitch0
# vim-cmd hostsvc/net/portgroup_set --nicteaming-policy=loadbalance_ip vSwitch0 

I was working with the ESX and ESXi and I was in for a shocking revelation... When I connected the ESXi ports which hosted the Managment network (even after making the Mgmt network to route based on IP Hash), the LACP bundle came up but there were 60% packet loss. On trying to find the reason, I found a weird work around.

Now, in my case, there is just one single subnet, so obviously, on the Juniper end, I had set to port mode access. Turns out ESX doesn't like it, though it is an access port, so to make the LACP bundle work, I converted the ae interface to trunk and set the access vlan as a native-vlan-id ... wallah !! it started working :)

Juniper EX configuration for the ESXi with Management Network

unit 0 {
    description "ESX1 - LAN";
    family ethernet-switching {
        port-mode trunk;
        native-vlan-id server_vlan1;

Now, if you are using multiple vlans, then you may not have faced the problem in the first place as the port mode would have been trunk. 

Another thing, I would like to mention is that, in Juniper, if you have noticed, I haven't put anything under vlan members, that is because, if the port mode is trunk, anything that you put under vlan members becomes tagged ... Wierd I know ...

I wouldn't leave this to chance that it is working, so i did some more digging around, I again read this link. (Which I had read some 5 times before), but here is what i read ... 

"HP Switches Sample Configuration

The following configuration is specific to HP switches:
  • HP switches support only two modes of LACP, ACTIVE, and PASSIVE while ESX/ESXi does not support either LACP mode currently.
  • Set HP Switch port mode to TRUNK to accomplish static link aggregation with ESX/ESXi.
  • TRUNK Mode of HP switch ports are the only supported aggregation method compatible with ESX 3.X NIC teaming mode IP hash. "

Now, if you will notice, the Juniper also supports only Active and Passive LACP modes ... I had ignored it as it was for HP switches and for ESX3.x (I am running 4.x), but I guess this is the only explanation as to why it works this way. 

Hopefully, you save some time digging around the info :)

NetApp and Juniper EX

NetApp supports LACP, so here we will see how to create a LACP vif on the NetApp and the corresponding configuration on Juniper EX.

In this case, on the netapp, we have 2 interfaces e0a and e0b (vif1), and on the Juniper side we have ge-0/0/2 and ge-0/0/3

On Net App:

If the vif is created, we will remove that and recreate.

the commands for NetApp are

1: ifconfig vif1 down
2: vif destroy vif1
3: vif create lacp vif1 -b ip e0a e0b

So, this is really very clear, we turned down an interface, then destroyed its configuration and then we said, we will create a virtual interface called vif1 with lacp and the load balancing method of “ip” , that’s what the “-b” switch stood for, and then we added the 2 interfaces.

The LACP is only available from version 7 and up, if you have a lower version, please go ahead and create a multi-mode vif and the Juniper switch configuration will like the ESX.

On Juniper EX

  • Create the ae interfaces as mentioned in the beginning (if you already have configured them, then you don’t need to do it)
  • Clean the configuration of the 2 interfaces.
The configuration on the EX switches will look like this

1: ae1 {
2:     aggregated-ether-options {
3:         link-speed 1g;
4:         lacp {
5:             active;
6:             periodic slow;
7:         }
8:     }
9:     unit 0 {
10:         description "NetApp1 - Controller 2 - LAN";
11:         family ethernet-switching {
12:             port-mode access;
13:             vlan {
14:                 members vlan1;
15:             }
16:         }
17:     }
18: }
19: interfaces
20: {
21:      ge-0/0/2 {
22:         ether-options {
23:         802.3ad ae1;
24:     }
25: }
26:     ge-0/0/3 {
27:     ether-options {
28:         802.3ad ae1;
29:     }
30: }

Once this is created, you can see the only difference is the LACP configuration that we put on the ae interface (this was not done for ESX/ESXi)
You can check if the interfaces are up, the very same way, but in addition, you also will have a LACP command to check.

1: root@PDX-SWIT-SVC02> show interfaces terse | match ae1.0
2: ge-0/0/2.0 up up aenet --> ae1.0
3: ge-0/0/3.0 up up aenet --> ae1.0
4: ae1.0 up up eth-switch

6: {master:0}
7: root@EX4200> show lacp interfaces ae1
8: Aggregated interface: ae1
9: LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
10: ge-0/0/2 Actor No No Yes Yes Yes Yes Slow Active
11: ge-0/0/2 Partner No No Yes Yes Yes Yes Slow Active
12: ge-0/0/3 Actor No No Yes Yes Yes Yes Slow Active
13: ge-0/0/3 Partner No No Yes Yes Yes Yes Slow Active
14: LACP protocol: Receive State Transmit State Mux State
15: ge-0/0/3 Current Slow periodic Collecting distributing
16: ge-0/0/3 Current Slow periodic Collecting distributing

You should be able to see if the LACP should be active for both partner and the Actor, if so, then the configuration works.

Hope you have found the guide useful. Please do comment if some thing needs to be corrected / altered.


  1. Another thing, I would like to mention is that, in Juniper, if you have noticed, I haven't put anything under vlan members, that is because, if the port mode is trunk, anything that you put under vlan members becomes tagged ... Wierd I know ...

    I wouldnt say weird so much as "yeah, that makes perfect sense, logically". If the port mode is trunk, then the VLANs you specify are "trunked" aka "tagged". I'm not sure what other behaviour you could have possibly been expecting here, honestly.

    1. Please read the full comment, I mentioned that way, you add a native-vlan-id in the switch and the port mode trunk and include that, then it will be tagged.

      In the case of HP or Cisco, when the untagged vlan is mentioned that takes precedence when compared to the the tagged part. Obviously Juniper needs to correct this and I guess in the coming code versions it should be corrected. so till that time, that is wierd, cause we have say "all" vlan members as trunk except vlan 2 , and so we add it in the native-vlan-id, yet, the switch will tag it, as it gives more precedence to the the members than to native-vlan id

      Thanks for reading ....

    2. "Please read the full comment, I mentioned that way, you add a native-vlan-id in the switch and the port mode trunk and include that, then it will be tagged"
      The native-vlan-id is not tagged tagged traffic, if this is what your statement is implying. That statement allows the trunk to carry untagged traffic, in your case the native vlan is server-vlan which is untagged.
      I would like to understand a little more of your comment but I am not clear on the second part. What are you wanting to say about vlan members all except vlan2? The main purpose of the trunk port is to carry tagged traffic. Adding the native-vlan-id statement is to allow it to carry both tagged and untagged traffic.
      Your configuration on the switch seems...
      1: set interface ge-0/0/0 ether-options 802.3ad ae0
      2: set interface ge-0/0/1 ether-options 802.3ad ae0
      3: set ae5 unit 0 description "ESX - LAN Bundle"
      4: set ae5 unit 0 family ethernet-switching port-mode access vlan members vlan1

      ae5??? You only created 4 so you should only have ae0 - ae3.
      But lets assume that is a typo, the access port will carry untagged traffic by default which I guess is what is coming from the vmware. What if yoor statement had read vlan members server-vlan1? The configuration explanation are not that clear so hence my confusion. I am happy it is working, however I am wondering about your explanation of why one failed and the other worked.

    3. Hi Traffikator,

      Possibly, I have confused my readers :) . All I mean by the statement is that the "port mode trunk" command takes more precedence that the native-vlan-id command

      so, if you have

      native-vlan-id 5
      port mode trunk vlan members 1-10

      The port mode trunk will take precedence and vlan 5 will be tagged, this confusion normally happens when you want all the vlans to be tagged other than the native-vlan-id, then in cisco you would add all under the trunked vlans and then a vlan id in the native vlan

      But in Juniper, if you do, the native vlan will not take effect at all, becauase all the vlans will then be tagged

      I hope that clears it out.

      About the second point, in my production box, where i had, i have more than 4 ae's :), hence it was a typo (I copied and pasted it from the wrong interface).

      Let me know if you have other questions. HTH

  2. One correction you missed the device count
    root@EX4200#set chassis aggregated-devices ethernet 4
    correction ( of course your output shows it anyway)
    root@EX4200#set chassis aggregated-devices ethernet device-count 4

  3. Hi :) i have a bit complex question but I was wondering if you could assist. i'm trying to reproduce this network design with a 2200-VC (2 members) and a HA pair of NSA 2400's: (diagram @ page 7)

    The NSA's are suposed to do three things:
    Allow inbound access from ISP external network -> VLAN A
    Allow outbound access from VLANs B,D,E -> Internet
    Allow limited connections from DMZ -> VLANs B, D, E

    The Juniper EX (VC) should allow VM's from ESXi hosts to a) communicate with each other/inter-vlan routing, and b) connect some VM's directly to the ISP network (assigned public IP),

    so I am trying to figure out if I should use the NSA's in transparent/L2 bridge mode, or if I should keep them as the primary routing devices for the network. Any thoughts?

    1. Hello Guy in Black T-Shirt,
      Firstly, apologies for the delay, It was holiday season here in india, so I dint check the blog

      Here is what I would do... (I am going to assume IP subnets here )

      ISP Subnet :
      VLAN B:

      Now, All we have to do is terminate the ISP VLAN on the Load Balancer. The Virtual IP of the Load Balancer will be in 100.100.100.x and the pool members will be in 192.168.1.x (which is the IP of the VM's)

      Now in this architecture this is how the best performance willbe

      1. Get a Routed Virtual Interfaces on the EX switches for VLAN B, C, D, etc - This automatically will get the Inter VLAN routing up. Please remember, this is better becuase if you made the 2400's as the default gateway the Intervlan traffic would also go there, and it will become the bottle neck.

      The inter - vlan routing will have absolutely no view of the 100.100.100.x subnet as they will not be on the switch. (If you have to put it on the switch, put it on another isolated Virtual Router)

      So, now we are left with the ISP access (Inbound and outbound)

      The Load balancer will accept the traffic on its virtual IP and send it to the pool members on the interface VLAN B. You may have to enable source nat if the outbound gateway is other that the 2400's. (But looking at the diagram, it doesnt seem so)

      1. The EX switches will point the default gateway as the common IP of the 2400's
      (in this case, SNAT is not required)
      2. The 2400's will have a hide source nat (PAT) for traffic originated from within other VLAN's (and also access rules) - This is for outbound access of all the VLAN;s for updates, upgrades, etc
      3. The Listener (Virtual IP) on the load balancer firewall will take care of the
      inbound access.

      Hope this helps

  4. having connectivity between citrix xenserver to juniper ex4500 with lacp configuration. and whenever xenserver transfer the data within chassis through that aggregated port but the speed is very low and i check the port end there were normal traffic flow.

    So the xenserver does not support LACP, please go through the below link.

    We need to configure the loadbalance on juniper ex4500 on that aggregated port.

    Reaquest for need to how to load balancing on aggregated port

    1. Hi Sameer,

      I see ... the only way around this is to use the Bonding without ether channel. Some thing that i have mentioned in this blog under the heading "ESX and Juniper EX"

      It is creation of the aggregated interface without the LACP

      1: set interface ge-0/0/0 ether-options 802.3ad ae0
      2: set interface ge-0/0/1 ether-options 802.3ad ae0
      3: set ae5 unit 0 description "Xen Server - LAN Bundle"
      4: set ae5 unit 0 family ethernet-switching port-mode access vlan members vlan1

      This should most likely solve this, but as i read from the documentation that you have provided - The below may be a worst case scenario....



      Is an active/active mode, but only supports load-balancing of virtual machine traffic across the physical NICs. Provides fail-over support for all other traffic types. Does not require switch support for Etherchannel or 802.3ad (LACP). Load balances traffic between multiple interfaces at virtual machine granularity by sending traffic through different interfaces based on the source MAC address of the packet. Is derived from the open source ALB mode and reuses the ALB capability to dynamically re-balance load across interfaces.

      In that case, you can configure the two ports as normal access ports and configure the Xen for balance-slb ... This will perhaps make one link behave in a unidirectional mode and the second in the bidirectional mode, nevertheless will have some load balancing and failover

      I will emulate this in my lab and let you know how this goes, or you can let me know how it goes for you