How To: Setup Cisco EtherChannel with ESX Server

In this blog, I will go through on how to setup a Port-Channel in a Cisco Catalyst 3750G switch, and setup that port-channel (etherchannel) to work properly with ESXi Server version 5.5. In my environment, it took, much, much longer to get this running because I had to completely re-architect my network to function this way. But if you’re building an environment from scratch, then this should be pretty easy to do.

I’ve verified that this config will also work with other Catalyst switches (2960’s, 3500’s, 3700’s, 4500’s, and 6500 series switches). This configuration will NOT work with Cisco Nexus switches, because the Cisco Nexus switches have different command line parameters than their Catalyst cousins.

So, let’s get going here.

I’m going to start by configuring a Port-Channel on my Catalyst switch.

interface Port-channel2
description Port Channel interface to DL380 Server
switchport trunk encapsulation dot1q
switchport mode trunk


After you create your port channel, you need to add switch ports to that port-channel. See below, as I add 8 ports to this port-channel.

interface GigabitEthernet4/0/11
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet4/0/12
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet4/0/14
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet4/0/15
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet4/0/21
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet4/0/22
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet4/0/23
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet4/0/24
description Port-Channel group to DL-380 ESXi Server
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on


From here I need to mirror the same VLANs that the ESXi server will have on it. So, lets create 5 VLANs to start. We can add more at any time.

interface Vlan1
no ip address
!
interface Vlan10
description Outside zone between pfSense and ASA
no ip address
 
interface Vlan20
description Inside network
ip address 192.168.1.2 255.255.255.0

interface Vlan30
description Front DMZ for direct connections from the Internet
no ip address

interface Vlan40
description Back DMZ -- Teired DMZ for server systems
no ip address

interface Vlan50
description Wireless network
no ip address


Now that the Cisco Catalyst switch is configured, let’s log into ESXi vSphere Client and configure the server to communicate with our switch.


I actually just bought a new quad port network adapter off of eBay just for this project. So after I installed it in my HP DL 380 G6 Server, I went in to verify that the card worked. And from this screenshot, it looks like it is working just fine.

esxi-1


Now go over to the “Networking” section. You can see I already have multiple vSwitches defined for my other 4 port network card that was already installed in the server. What my plan is going to be, is that I want all eight of my network adapters to be part of one port-channel. This will maximize the throughput and bandwidth to and from the server, as well as provide a reliable 8 way path to my core switch. The only downside to this is that my core switch is now my single point of failure on the network. I recommend that if you’re going to do this in your environment, you should have an identical switch and a full backup of the configuration on your primary switch so that you can swap out if the primary fails.

esxi-2


From here, click on “Add Networking…”

esxi-3


You need to select “VMkernel” here. You’ll be using “Virtual Machine” network type later. For now, VMkernel, then click “Next”:

esxi-4


Select the network adapters you want to participate in the port channel, then click “Next”:

esxi-5


Since this is a Port-Channel, or Etherchannel, you want this to trunk all of your VLANs from the Cisco Catalyst switch to your ESXi server. Make it easy and name this “Port-Channel” and allow all VLANs to traverse the link, then click “Next”:

esxi-6


You’ll want to enable management on this, so give it an IP address on your Internal network. Please, for the love of all that is right and just, do NOT open up management access to the Internet or any of your DMZs!

esxi-7


Verify your settings on the “Summary” screen, then click “Finish” to continue.

esxi-8


After you create your switch, you’ll see it appear in the “Networking” screen of your vSphere client. You’ll see that I haven’t attached network cables yet, which is why all my adapters are showing as “Down” with the red “X” next to each physical adapter.

Go ahead and click on “Properties…” to continue.

esxi-9


Make sure your vSwitch is highlighted in the left column, then click, “Edit…”

esxi-10


In the vSwitch Properties window, make sure that you have ESXi “Route based on IP Hash”, then click okay.

esxi-11


Now you can add in all your VLANs that will live on this vSwitch. So click on “Add Networking…” to continue:

esxi-3


Here is where you’re going to use the “Virtual Machine” connection type. Click Next to continue.

esxi-12


We’re going to bind this VLAN to the new switch we created. So select the vSwitch you created earlier in this process, then click “Next” to continue.

esxi-13


Here, I will create a Business-to-Business VLAN, and I’ll tag all traffic in this VLAN to #75. Then click “Next” to continue.

esxi-14


Verify your changes in the “Summary” screen, then click “Finish” to continue.

esxi-15


After you create all of your VLANs and add your virtual machines to each network you desire, your end result will look like this:
esxi-16



If you have any questions, please feel free to contact me at any time!

http://vmwaremine.com/vmware-vsphere-best-practices/
http://vmwaremine.com/2012/05/29/networking-configuration-for-esx-ot-esxi-part-3/
http://frankdenneman.nl/2013/01/28/vmotion-and-etherchannel-an-overview-of-the-load-balancing-policies-stack/
http://www.virtualizetips.com/2011/03/05/esxi-management-network-issues-when-using-etherchannel-and-nic-teaming/
http://blog.scottlowe.org/2008/07/16/understanding-nic-utilization-in-vmware-esx/
http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048
http://www.sysadmintutorials.com/tutorials/vmware-vsphere-4/vcenter4/network-teaming-with-cisco-etherchannel/
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003825
http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1010778
http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1003825
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003806
http://searchnetworking.techtarget.com/tip/How-to-configure-Virtual-Switch-Tagging-for-vSphere-VLANs
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003825
http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/
http://www.sysadmintutorials.com/tutorials/vmware-vsphere-4/vcenter4/network-teaming-with-cisco-etherchannel/
http://serverfault.com/questions/628541/esxi-5-5-nic-teaming-for-load-balancing-using-cisco-etherchannel
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001938
http://www.simongreaves.co.uk/vmware-nic-trunking/
http://www.geekmungus.co.uk/vmware/vmwareesxi55managementnetwork-nicteamingandvlantrunking
http://www.mustbegeek.com/configure-nic-teaming-in-esxi-server/
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004074
http://www.ahmedchoukri.com/?p=298
https://glazenbakje.wordpress.com/2012/05/10/cisco-catalyst-switch-ether-channel-settings-to-vmware-esxi-5/
http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048
http://longwhiteclouds.com/2012/04/10/etherchannel-and-ip-hash-or-load-based-teaming/
http://wahlnetwork.com/2012/05/09/demystifying-lacp-vs-static-etherchannel-for-vsphere/
http://www.amirmontazeri.com/?p=18

VN:D [1.9.22_1171]
Rating: 5.0/5 (2 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

AT&T u-Verse Static IP work around with pfSense

First off, I’d like to give AT&T an honorable mention (sarcasm) for using the fucking worst, P.O.S. garbage, DSL Modems on the planet: 2WIRE. These things are ridiculous. You’d think that if a provider was able to route a /28 subnet to your home/business, that they’d be able to properly manage that subnet through their “firewall” or whatever you want to call it. The way this normally works is through routing a network range to your device. But AT&T and 2WIRE ensure that for every public static IP address you have, it has to have a unique MAC address and it must look like a different device all together. This is asinine.

So, with the help of my business partner, we’ve come up with a solution on how to get a set of static IP addresses to work so that you can host services on AT&T u-Verse. The way we accomplished this was through the use of an open source and free operating system named, “pfSense”. I’m sure there are other systems out there that we could have used, or just done it in Linux, but pfSense is really robust and has a nice interface. So that’s what we went with.

Additionally, I’m sure not everyone and their mother have an HP DL380 running in their basement, but… welcome to the Erdmanor. I have a DL380 in my basement. So what we’ve done is virtualized a firewall. We’re running pfSense in a virtual machine on the DL 380, which is running ESXi 5.5. I know ESXi 6.0 has been out for a few months now, but to be honest, I’m just too damn lazy to upgrade my box.

Anyways, here’s how we configured the virtual firewall. In ESX, we provisioned the system to have 8 network adapters, a 10GB HDD, 2GB RAM, and 1 virtual CPU. From there we added the VM to access the three different network segments (DMZ, Internal, Outside), and created the interfaces within pfSense. Then we programmed the AT&T gateway to use the external addresses that were provided by them, making sure that the proper interfaces and MAC addresses lined up between the ESX server, the AT&T gateway and the pfSense console. Also, in the AT&T gateway, we setup the system to be in DMZplus Mode, which you can read about in the screenshot below.

pfSense1

pfSense2

pfSense3

att-config0

att-config1

att-config2

att-config3



Now that our AT&T gateway is properly forwarding External IP traffic to the proper interfaces on our pfSense firewall, we can go through and create all the inbound NATs, firewall rules and network security that we wish to have.

If you have any further questions on how to set this up, just ask!

Thanks!





VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Serious network architecture that works for everyone.

I started writing this blog post as a way to setup a reverse proxy for mail inspection, but it turned out that a network architecture blog focused on security of the perimeter was more important. I’ve gone over in my head with all the companies that have told me, “Ohhh we don’t need this” or, “this is too much administrative overhead” or, “We don’t need this much complexity, we’re just manufacturing “widgets” , or something like that. And to those people, I say this: “I am so sick and tired of hearing excuses of why you think it’s okay to be lazy. Do it right, do it now, and save yourself the headaches of a breach.” We’ll talk about the costs associated with being penetrated some other time, but it’s EXPENSIVE!

If you’re planning to do this right, then you’ll want/need to have a multi-tier DMZ for your public facing services. We’re not talking about internal servers or your internal network at this point (though after thinking about it, the exact same concept can be carried out on the Internal network too). In this blog, I’m merely trying to tell you about your externally facing services. This blog will go over proper placement for Internet facing services. The VAST majority of companies out there don’t go to this level of sophistication, but its totally possible for any company to do this and if you really want to secure your network infrastructure, then you’ll at least attempt this.

Before we start, I’ll say this. I’m going to try my best to describe this as granular as possible. There are a TON of intricacies here that need to be thought out. I’ll provide a rudimentary Visio diagram to help on this, but you’ll need to map out your own network and break it down in a way you can understand.

The main point of network segmentation and building a secure network architecture is based on one of the most talked about security areas: The Principle of Least Privilege. Do your end users need access to databases? How about other network services? How about filesharing with eachother? How about shared resources for just once specific department (should engineering folks be able to communicate with financial systems)? Please think about the level of access people should have to services while going through this blog.

You’re first level DMZ should house only your front end web servers (or load balancers in front of those servers), DNS servers and your proxy servers, nothing more. These systems are extremely visible to the public and will be processing thousands of requests per day, so if anything happens to them, trust me people will notice. Remember, these are front end systems, so you don’t need much of anything out there. I’ll be going over how to set those services up on a future blog, but for now just remember, least privilege. Internet users dont need direct access to the webserver, they do need access to a reverse proxy server that inspects the traffic going to the web server(s).

From here, you can create your second tier that will house your web servers (if you have load balancers or proxy servers in front of them out in tier one), mail servers, SFTP servers, if you’re using LDAP or AD you could add a read-only domain controller (RO-DC) for authentication (but NO Internet access), and things like that. These systems should be using local firewalls as well as network layer firewalls to control access to them. Web servers dont need to talk to anything except the back-end SQL BD and the end user. Both of those firewalls should specify that the only systems they’re allowed to talk to is the server in tier 1 that proxies data to it, and if there is a server behind it in tier three.

Then there is an optional third tier where you would house your back-end database servers and any other servers deemed unnecessary for tier two. when I say optional, I dont mean just throw it away and put SQL over in tier 2. I mean, if you dont have a SQL back-end you can eliminate tier 3. Another RO-DC could be posted here for authentication services(again, NO Internet access!). If you’re running MS-SQL or Oracle SQL servers here, you can have services level authentication (or any other services for that matter) authenticating to that RO-DC. Same goes for tier two.

Lastly, I’ll mention a Management network that will have access to all three tiers. You’ll obviously have admins (even if it’s just yourself) that will need to run updates on those boxes or perform other administrative functions on those systems. Don’t forget to allow yourself access to that. But that doesn’t mean “IP ANY ANY” from the management network into those tiers either! Dont be LAZY, be smart and do it right.

In my Visio diagram, I used some old hardware, and multiple physical switches, but don’t forget, you can trunk VLANs and do some pretty cool configurations with Cisco gear, especially the new ASA’s. See it here: DIAGRAM LINK

So from tier 1, your DNS server should only be servicing requests from your 3 DMZs, and the Internet. I would say, you shouldn’t open this up to your internal clients, because you should already have internal DNS servers for Active Directory (or what ever LDAP service you’re using). At most, you should only be allowing 53/udp inbound to that DNS server from the Internet, and allow SSH inbound to that server from your management network. That’s IT! For your proxy server or load balancer, you should allow 80/tcp and 443/tcp inbound from the Internet and allow in whatever port your load balancer needs from the management network. So in this scenario, you should have 80 and 443/tcp and 53/udp open from the Internet to tier 1. Simple, see?

Tier two only has ports open FROM tier 1 into tier 2 (and management network into tier 2). The people out on the Internet will NEVER communicate directly with tier 2, there’s just no need.

And lastly, tier 3 will only accept communications from tier 2 and the management tier. No end user needs to communicate with the SQL DB’s directly, so why let them?

The only thing I’ve left out here is the RO-DCs. What do they communicate with? Well, the way I would set them up is have the 2 real Domain Controllers in the management network. This should be a totally different domain than your internal network. Name the domain whatever you want (fubar.dmz or whatever). Your RO-DCs are only acting as a proxy to the domain. Nothing is stored on them, so you’ve got really nothing to lose.

So that’s really about it. If you’ve got any questions, contact me via my LinkedIn profile. There’s a link to that right on my home page near the bottom of the left column.

 

Enjoy!! 🙂

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)