20170217

SYNPROXY To The Rescue: PFSense TCP Solution for ERR_CONNECTION_TIMED_OUT

Objective:

  • Establish TCP connection to a server in a different subnet and interface using a PFSense Firewall.
  • Allow for SSH, HTTP and HTTPS connections from a LAN client  to a FreeNAS box.
  • Keep mind intact and don't lose any sleep.

Summary:

  • If you are experiencing slow connections that never seem to complete the TCP handshake, make sure your firewall and the rules/chains are correctly configured for the type of connection you are attempting.
  • After a couple of hours (to be honest more like 4 hours or troubleshooting 1 hour of blogging), I couldn't figure out why I kept receiving TCP TIME_WAIT and CLOSE_WAIT for a simple HTTP/HTTPS connection to my FreeNAS box.
    • Simple topology
      • 2 interfaces and 2 networks.
      • Create a rule to allow for network A (LAN) to talk to network B (FreeNAS).
      • TCP connections from a LAN client to FreeNAS box.
    • Complicated results
      • Created a simple rule for LAN to FreeNAS with Any Protocol, Any Source, Any Destination, and Any Port.
      • Unable to connect to FreeNAS via HTTP/HTTPS
      • Successfully able to connect via SSH.
      • Successfully able to ping between LAN client and FreeNAS box.
      • Unable to connect to FreeNAS HTTP/HTTPs port over netcat:
        • netcat <FreeNAS IP> 443 
      • OK check the logs, seeing a lot log entries for ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, and TIME_WAIT from LAN client to FreeNAS box.
      • Check chrome and inspect the traffic, same info as the logs. Good.
    • Identified the issue, but don't understand it enough
      • At this point I can see that there is a TCP connection issue, the connection goes as far as ESTABLISHED but there is an issue once the LAN client goes into the FIN_WAIT_1 state.
      • Now the problems start, no ACK from FreeNAS box and LAN client goes into FIN_WAIT_2, again no ACK from FreeNAS box  now the LAN client goes into TIME_WAIT.
      • At this point while client is in TIME_WAIT, any other attempts to connect to the FreeNAS box are going to be blocked because now the wait time needs to expire.
        • The wait time is calculated as 2 times the Maximum Segment Lifetime
        • MSL on PFSense 30 seconds so that means 2*30 = 60 seconds
          • Use the following command to determine MSL on PFSense
            • # sysctl net.inet.tcp.msl
            • output > net.inet.tcp.msl: 30000
      • After TIME_WAIT expires, the connection goes into CLOSE_WAIT and browser reports ERR_CONNECTION_TIMED_OUT
    • Solution
      • I didn't have one at first and could not determine how to solve this problem.
      • Early in the day I recalled an Advanced Feature inside the PFSense Rules from  PFSense documentation about Asymmetric Routing and Firewall Rules.
      • There is an option to select the handling of State Type to be used for the firewall rule.
        • State types - the pfSense software offers multiple options for state handling.
          • Keep state - Works with all protocols. Default for all rules.
          • Sloppy state - Works with all protocols. Less strict state tracking, useful in cases of asymmetric routing.
          • Synproxy state - Proxies incoming TCP connections to help protect servers from spoofed TCP SYN floods. This option includes the functionality of keep state and modulate state combined.
          • None - Do not keep any state entries for this traffic. This is very rarely desirable, but is available because it can be useful under some limited circumstances.
        • The Keep state is the default set to the rule used in this project, but as far as documentation and what this state does, I found nothing useful. 
        • Fortunately Sloppy and Synproxy states are documented extensively and I was able to determine a solution now.
        • The Synproxy state handling is the solution used to sucessufully establish HTTP/HTTPS connections to the FreeNAS box from the LAN client!
          • Here is quick summary of what Synproxy does:
            • The LAN Client connects to the Synproxy server (PFSense box).
            • The Synproxy server estbalishes the TCP connection to the FreeNAS box on behalf of the LAN client via spoofing LAN client's SYN responses.
            • Synproxy server and FreeNAS server continue the handshake and once a connection is opened to the FreeNAS box, the Synproxy server hands it off to the LAN client to communicate with the FreeNAS box.
          • The downside
            • The PFSense box becomes a proxy for these TCP connections and this could have potential performance impact in much larger environments.
      • Overall, a terrific learning experience and really good knowledge for me to share to all of you. Thanks.


    Really Really Good References
    https://www.pfsense.org/about-pfsense/features.html
    https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules
    https://en.wikipedia.org/wiki/Maximum_segment_lifetime
    https://benohead.com/tcp-about-fin_wait_2-time_wait-and-close_wait/
    http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html
    http://rhelblog.redhat.com/tag/synproxy/
    https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY
    https://www.cyberciti.biz/tips/freebsd-lists-open-internet-unix-domain-sockets.html

    20170215

    10GbE LAN: FreeNAS, VMware ESXi, Windows 10 and $100 Budget

    Objective:
    1. Create a 10Gb Ethernet LAN for the purpose of...I have no real purpose.
    2. Spend no more than $100.
    3. Only 3 computers will be on 10GbE but can only use 1 PCIe x16 port (version 2 and 3).
    4. Use fiber to connect to 1 server not collocated with other 2 systems.
    5. Dual port NIC supported by FreeNAS, VMware ESXi, and Windows 10.
    6. NIC must be able to support a variety of SFP+ Multi Mode and Single Mode optics.
    7. Nice to have: MacOS and Linux support over thunderbolt adapter(future projects).
    8. Ideally would like to have NICs with well documented support and use.
    Summary:
    • After a week or two of ebay searches and research on the internet, I came across a few NICs 
      • Mellanox ConnectX (affordable, but not supported FreeNAS)
      • Chelsio T Series (pricey, 100% compatible)
      • Intel X520 (expensive, unable to determine FreeNAS support) 
      • SolarFlare (best price, 100% compatible)
    • Linux and Windows support for all NICs listed above were highly likely but the FreeNAS support was the issue
      • https://www.freebsd.org/relnotes/9-STABLE/hardware/support.html#ethernet
    • Chelsio is the most common choice for FreeNAS builds, it is listed in the hardware requirements and mentioned a lot in the FreeNAS forums.
    • Decided to go with Solarflare:
      • Solarflare is not mentioned very much in FreeNAS forums and I was almost hesitant to use these NICs because of the lack of documented use I was able to come across. The deciding factor was a youtube video and blog I came across by Scott Schweitzer
      • I was immediately sold after looking at the list of systems with driver support  for Solarflare NICs.
        • Systems Supported
          • Apple Mac OS X
          • FreeBSD
          • Linux
          • Solaris
          • VMware
          • Windows
          • XenServer
      • Another plus is that HP uses Solarflare adapters.
      • Potential Prices of used Solarflare adapters
        • Here are the different series of adapters available
          • SFN5000 (Best Price, 100% Compatible) Around $30-$50
          • SFN6000 (Good Price, 100% Compatible) Around $40-$80
          • SFN7000 (Over budget) Over $100
          • SFN8000 (Over budget) Over $100
      • Purchased three SFN5000 series dual 10GbE NIC for a great price of $80 USD total! 
        • This purchase came with four SFP+ MM Optics.
        • Tested and updated all NICs with the latest firmware
      • Last purchase was for two Cisco Twinax SFP+ DAC cable for $20 USD.
    • So with some luck and lots of research, I was able to deploy a 10GbE LAN for $100 USD.  Enjoy!

    20150811

    Matt Horne

    Indoc, Winter 2005

    Horne, Sabel, Huff, Grant and many others were indoctrinated into the 3rd Battalion 2nd Marines Scout Sniper Platoon. I recall the favorites, the candidates who people wanted to see make it to the platoon and the one’s people wanted on their team. During this time I was a team leader in the platoon and I didn’t think I’ll be going back for another tour. Well as many Marines know, in the Marine Corps, anything can happen.

    The Draft

    I don’t recall how we (Team Leaders) decided who had first pick, but what I do recall is that I picked individuals that were different and capable of anything I could throw at them. These guys were the underdogs on that draft list, but they were guys I wanted to live and die with. Nine years ago seems like a long time and trying to remember the type of person I was then, is fairly difficult. From what I recall and what many have said to me, I was an intense person and pretty much a pain in the ass to anyone who worked with me during those times. So for the sorry bastards who were to become my teammates, life had in store for them things that no one could have imagined.

    Reaper 4: Mendez, Horne, Sabel, Huff, Grant

    The first run is always the hardest, for everyone. In a team, everyone possess at least one strength and at least one weakness. When faced with many strengths and many weaknesses, leaders should learn quickly what works and what doesn’t, to bring a team together. We ran to break down our weaknesses and build up our strengths. As we prepared for our first run, one could sense the anxiety, fear, and excitement emitting from each of us. I’m not sure where it came from but I know that it didn’t help to have a team leader like me, one who was rumored to have ran someone into hyperthermia in the past (sorry about that Sanchez). Nonetheless, we set off on our run. Through the quad and across the road, we ran with no objective and no end in sight. What the team didn’t know at the time, we were already beginning our first training lesson.

    “Contact Front!”

    Many teams that morning ran and pushed the physical limits of their teammates. Breaking them down to build them back up. As we sprinted to the running path near the bay, I could see the fierce concentration and focus each team member had when I looked at them. It was the perfect time to call an audible and break down the first obstacle, misconceptions. On a combat patrol, each team member is dependent of the man in front or behind them. There is always some method of communicating to everyone about what is going on. In the event someone spots the enemy, and if the enemy has spotted you, several things can take place: inform your team; take cover; or engage the enemy. By calling “Contact Front” in the middle of a run, I tested each member for their reaction.

    Lesson 1: Brother’s

    All these years have gone by and I never forget how important that first lesson was for both my teammates and myself. We all learned something about each other, we all learned what the expectations were going to be and how we were going to operate as a team. Horne was the only other team member, besides me, who had deployed to a combat theater. Sabel, Huff, and Grant were fresh out of SOI (School of Infantry). Horne would be my Assistant Team Leader but no matter what, we were all equals and treated each other as such. Each man’s life mattered just as much as everyone else’s.

    Horne

    Horne spoke about his personal life in Ohio, that life we all had before the Marine Corps. One time he pulled out a dental hygiene kit that looked like something out of a thriller movie . Horne use to work in a dentist office and became a sort of a neat freak about his teeth. So everywhere he went he had this hygiene kit, so he could give himself a deep cleaning of his already near perfectly white teeth. Horne needed something to keep him focused and stimulated. This was his thing, everyone had something they were obsessed about and this was his. Now that we were a team, STA and Reaper 4 was his life. For Horne, these were the things that made him happy in that time, the things that helped him move on from his past.

    Goodbye

    That first run with you and the team was the first memory that sparked into my head when I heard you past away. After you survived being shot at in Iraq, the pain you suffered afterwards just made it harder for you to live a normal life. But you tried everyday and I always wished you the best. I’m really sad you’re gone brother, but it makes me even sadder to say goodbye. I’ll see you soon Matt, Semper Fi.


    20150706

    Using Rsync To Backup Directories



    Here is a quick command to backup a local directory to another local directory. For this example, the Rsync command will conduct a dry run for a backup of an entire USB drive to a directory in another USB drive on a Debian system.

    ---

    haventfoundme@debian:~$ rsync -a --dry-run -v /media/haventfoundme/school/ /media/haventfoundme/backups/USB_Drives/backup_of_school_drive
    sending incremental file list
    ./

    sent 247,146 bytes received 1,508 bytes 45,209.82 bytes/sec
    total size is 64,833,623,364 speedup is 260,738.31 (DRY RUN)



    ---


    The above example shows that all files and directories from the source, "school" usb drive, are synced (backed up) to the destination directory (backup_of_school_drive). Also keep in mind the forward slash on the source and not on the destination path.

    Below will show an example when a new file is created in the source and detected during a dry run.

    ---

    haventfoundme@debian:~$ rsync -a --dry-run -v /media/haventfoundme/school/ /media/haventfoundme/backups/USB_Drives/backup_of_school_drive
    sending incremental file list
    ./

    test.txt

    sent 247,170 bytes received 1,511 bytes 55,262.44 bytes/sec
    total size is 64,833,623,364 speedup is 260,709.00 (DRY RUN)

    ---

    Notice that the dry run shows the exact file which was created in the source and does not exist at the destination. Also notice the amount of bytes received has increased slightly.

    Once you are ready to sync your files over, remove the "--dry-run" switch and run the command again. Enjoy!




    20150618