Router/Firewall/Captive Portal Wifi Project

[Presler]

Right, the history behind it all.

3.5 Years ago, I started working for a hotel in Sandton.

When I started working there, the place was in a dire situation. The local LAN was in a state where everyone would hope by arriving for work the next day, everything would still be working. The AD/DC and file storage server was a entry level blade with 2 x 250Gb RAID 1 setup, one drive dead and a lost admin password. Took us several months to basically rebuild the entire setup, new HP DL380 servers and brand new domain.

Part of this dire situation was a mess of 2 somewhat functional Wifi networks. One was ran over 3 x 4Gb ADSL lines bonded(very reputable IT Company is SA), the other one was sitting on a 10Mb Fiber line(small 5 people home office type setup). Both places utilize configurations of large outdoor antennas placed on edges of 50m buildings. 2 Problems here. Rooms in the center of the wall has little to no signal because of distance/tonnes of concrete in the way, secondly 2 antennas covering 50 odd rooms per wall creates huge congestion issues. Our complaints lists was massive. So to the drawing board we go and decide it's time to part ways with both. BTW, 10Mb Fiber is our own, which at the time we upped to 30Mb.

I suggest a setup of strategically placed switches hardwired to oodles of AP's, placed in such a manner that no matter which direction you go, 8m from one AP is the next one. Coverage sorted. Signed off, I am ready. Through other daily tasks 6 months later I laid 4.5km of CAT5 cable, mounted 110 PoE AP's, 6 x 9U cabinets with switches, patch and brush panels & UPSed for Loadshedding/Generator on time cover. We pull in a suggested company to handle the routing/billing side for us. Things go well, until this company's abilities fail dismally and our complaints start to rise. At this point i have resigned at the company, started my own business and they now become my client. Someone clever decides best way forward is to buy a Mirkotik router and have that programmed with the 5 people home office guys to handle the routing etc. I arrive one day at the client, get the router stuffed in my hands to make due. What a nightmare. I don't doubt Mikrotik's abilities, but damn their software is difficult to get your head around.

Last year our 30Mb starts hitting fair usage policies around the 14th of each month. December 2014 we pull out the stops and go 100Mb 1:1. Things go well for a while, until we start hitting fair usage policies again around 20th each month now.

Several weeks ago I'm tasked to look into the problem and fix it. Ever since I worked for them my plans were to bring all sides of IT in house. So I set my mind to it and go. Lots of reading and research. I learn in the meanwhile the Mikrotik is not able to handle the load it receives. This explains many issues. Over the years I've had meeting on meeting with all kinds of people, options are expensive. So I look into free ways. Some research brings me onto this list.

Captive Portal Software

With my experience in the matter, I'm sifting through and looking for easy options, skipping anything command line and looking difficult. I work into many of the options and for a while get stuck into Zeroshell. Now Zeroshell seems to have some features, but lacks one thing we really need, customization of the landing page where the guest would authenticate. Keeping in mind, I know nothing about web design either, so I'm really drowning here. Struggling with ZS for a while and on the side running 2-3 other options, I was looking for an image of something and I stroll across one of the options I skipped, pfsense. For some reason, everytime I come across pfsense I skip it because it looks pure command line. Until I actually start looking into it.

Just WoW. I figure out the little FreeBSD derivative uses the same interface and pretty much everything else Mweb uses for their Managed Internet Gateway, plus boat loads of other features. I download the 99mb package, grab a 4 NIC HP Blade server, drop in an SSD and install. 2 minutes later it boots from the SSD and I'm ready to go.

4 x 20 hour days later, lots of reading up, homework, testing, breaking it FUBAR and starting fresh I switch the Mikrotik off and pfsense on. 1 hour in I have 421 people online, user control and limitations in place all user browse happily with minimal input required to log on and best part of it all, my 100Mb is not abused any more, most traffic concurrently was pushing 11mb on the 100Mb line. Piggy's in cow poop can not be this happy.

During all this, I have learned much about routing, some bits about designing a webpage and putting it all together to make my previous work shine even more.

I urge you, you got an old PC sitting around doing nothing, drop an extra nic and pfsense in it and just start playing with it. Seriously powerful stuff even if you just use it to better maintain your home connections. on a side note, here is what I used, all software free entirely.

HP DL160 server(Xeon 5405 2.0GHz, 4Gb RAM), 60Gb SSD

pfsense

For the web page design, both packages a few years old, but did the job for me and should be good for any beginner, just like me.
NVu and Kompozer


Quite a bit of reading over the last 3 days gave me more options and ideas. Fluid web design, plug-inn's for pfsense to give it more functionality, heaps of more tips and tricks.

After I've had some sleep, I'll add another post with links and notes for anyone who would like to work on or try out a similar project.

 

[MeTsU]

Thanks for sharing - great going! Look forward to reading more.

 

[raydummer]

Great thread, Really enlightening ! - Hoping to read more


I do agree with you on the Mikrotik part ......There is SOOOO much potential and for the value it's a bargain. BUT the biggest downfall is that there is no source code of RouterOS for the community to improve upon .....aka like PFsense and so many.
I personally would love to see your workings in PFsense, etc. as I'm currently building my own little box.

 

[Phence]

December 2014 we pull out the stops and go 100Mb 1:1. Things go well for a while, until we start hitting fair usage policies again around 20th each month now.

This sounds like there is no rate limiting being done on the router. What one approach would be is to segment the network into different vlans, breaking up the broadcast domains. Obviously you would need managed switches for this. You could setup rate limits for different VLANs on the Mikrotik, even the entry level RB750 models can do this. Then set up some sort of monitoring (eg. Cacti) via SNMP on the Mikrotik. I found monitoring is an essential tool for getting an overview into problems.

Any device with a CLI can seem daunting, but once you understand the logic of the Mikrotik CLI it is quite simple. All you need to do from there is understand the networking terminology.

I am not sure if any ISP even offers a 1:1 service that is not uncapped? My understanding is that 1:1 is dedicated which also means uncapped. Even having a 1:1 service with a FUP is actually misleading to me. Were a handful of users just saturating the link? This is what often happens if no management is applied to internet connectivity, someone is torrenting or something else and no matter how fast you make the link, it is always too slow! This is why firewalls are a great way of managing the last mile, probably why you found pfsense so helpful.

 

[Presler]

This sounds like there is no rate limiting being done on the router. What one approach would be is to segment the network into different vlans, breaking up the broadcast domains. Obviously you would need managed switches for this. You could setup rate limits for different VLANs on the Mikrotik, even the entry level RB750 models can do this.

Oh for sure. One of my first considerations doing this was rate limits. I'm pretty sure there was no bandwidth control in place. I could speed test from a cellphone in any room and get 35Mb down and up no probs. What they did do was to use OpenDNS and block the silly shit out of everything, Porn, P2P, VPN, everything. This created some problems as some users mail servers stopped working etc etc. I was also sworn at daily by people not being able to view their porn. To some extent I can understand why it was done. For one reason at least, public area with no authentication open access internet, imagine a mother catching her 15yo surfing porn to hearts content. He'll be in shit, but so will we for allowing it since the network is open to anyone.

The other thing was that the router was handling 1 large network. 330 room hotel, [MENTION=32969]iNAtRUCKTOR[/MENTION] new favorite restaurant that easily seats 100+ people and 9 conference venues with capacity for 1000+ people. This is no dinky little router either, it's an RB1100AHx2, which through plenty of reading I figured out tops out at around 500 users. Part of this project was to not just go better but think smarter. So I split the network into 3 sub divisions as well, Restaurant, Rooms, Conference.

Restaurant gets 512k up and down per user. Might sound like little in today's terms, but consider what a restaurant goer does. Take a pic of the food and upload to FB, update Twitter, Check in on FB, perhaps look for a photo or something to show around the table. All of which 512k per device is enough.

Rooms. 1Mbps per user up and down. We sat in a room with an Iphone, Ipad, Android Tablet, Android phone and 2 laptop connected to 1 AP. All users had something going on, laptops a bit more. We had Youtube streaming a vid at 360p, 2 tabs on forums, 1 tab on FB scrolling, email open in the background downloading and uploading several mails. No issues.

Conference. 1Mbps down and up with login accounts for presenters with 2Mpbs up and down. This allows for Webex/Skype and the like to happily do it's thing while everyone else get a fair share.

Then set up some sort of monitoring (eg. Cacti) via SNMP on the Mikrotik. I found monitoring is an essential tool for getting an overview into problems.

Yes, I am investigating still into the monitoring side, not just what happens on the routers but also the switches and AP's. The two suggestions that came up is Cacti and The Dude. Both I know will happily work off the Mikrotik as well. However, for any remote assistance and to get to all 3 networks easy, I set up a networked PC that's hooked up to all 3 networks. From there it can check the firewalls and networks and report anything going on, for which I believe The Dude would be the better option. Still reading up about options though.

Any device with a CLI can seem daunting, but once you understand the logic of the Mikrotik CLI it is quite simple. All you need to do from there is understand the networking terminology.

If it all was MS back end, CLI wouldn't have seemed so daunting. But most all of the proper options including pfsense is Linux based. With pfsense, most of the thinking is handled for you. Even command line is made easy with "put your info and we will do the rest". Even so, I've learned much of the terminology already.


I am not sure if any ISP even offers a 1:1 service that is not uncapped? My understanding is that 1:1 is dedicated which also means uncapped. Even having a 1:1 service with a FUP is actually misleading to me. Were a handful of users just saturating the link? This is what often happens if no management is applied to internet connectivity, someone is torrenting or something else and no matter how fast you make the link, it is always too slow! This is why firewalls are a great way of managing the last mile, probably why you found pfsense so helpful.[/QUOTE]

The line is 100Mb 1:1 uncapped. One can imagine that the FUP on such a line would be something out of this world, which it should be. However, our most used ever month, we hit 19.2tb going through that line, so FUP can be expected. This exactly was the reason I considered restrictions rather than blocking. That being said, to improve stability even more I can run proxy services through Squid, URL filters through Squidguard and Intrusion detection through Snort.

Something to make you think a bit. Any hotel guest will gladly complain that internet is slow/buggy or rant about anything else they can find, not considering just how much time and effort(and money) is thrown into providing that internet or consideration for his/her fellow guest. This particular hotels Wifi is completely free, you don't pay a cent. On top of that, very few users rock up with one device. On one previous occasion, we got a call from a very irate guest. Hotel 100% full, he told us that both his and his wife's phones and tablets are online, but neither of their laptops will connect. That's 6 devices-1 room in a 330 room hotel. On top of that, most devices once connected starts pulling updates. Apple is any Wifi admin's worst nightmare. Every time Apple brings out new hardware or OS(which is every 6 months now) their ballgame changes. There was one stage 2 years ago where after such an update where every Apple in the building refused to connect at all. Took us 2 days of patching to just get them online. So just think about the saturation going on in the background with 800-1000+ devices wanting each to update/connect whatever else when they have no bandwidth control in place.

A mention of the hardware pfsense runs on.
Obviously, the smaller your hardware the less you'll be able to do with it. pfsense allows you to run separate Captive portals on each available nic you have. On the HP server I have 4 nic's, 1 for WAN, one for Rooms Backbone, 1 for Conference Backbone and one on my monitoring machine. pfsense reports average load of 0.09%. Which means I haven't even started to tap into the potential of pfsense and the server.

The Restaurant runs an average of 80 users concurrently. There is an E7300 PC with 4Gb RAM and SSD, pfsense reports 0.9 to 1.0% average load.

 

[iNAtRUCKTOR]

[MENTION=385]Presler[/MENTION] The restaurant is still one of the best I have gone to and what you have done with the network turned it into a more socialize and enjoyable environment, would recommend this restaurant to anyone within a heart beat, especially those guys that really want to go full out for a special lady.

 

[Presler]

Thanx buddy. Like I said to you before, that place is an always evolving project, always looking at better, easier, more user friendly and most of all fair towards all way of doing things. At the end of the day, everything we do is to make the experience for the guest better, restaurant or hotel.

Right,

On pfsense, the forums is full of info. Mikrotik has a Wiki, which their forum users are glad to point you to rather than sometimes just helping a guy out. The pfsense community is much more helpful at times giving away their own code for others to play with or use. Also, different to most of the Mikrotik guys, there guys try to explain most stuff in laymen terms, making it much easier to understand by the beginner.
pfsense forums

Easy info on limiting bandwidth per user in pfsense.
Limiting Users
In pfsense the captive portal allows you to also limit bandwidth per user, but there are tools available to bypass the captive portal and so bypass the limitation . The setup in aforementioned link initiates bandwidth control even before you get to the captive portal.

Info on the pfsense forum to install and setup Snort(network intrusion plug-in)
Snort

Info on installing Squid on pfsense(Proxy server)
Limiting Users on pfsense

Info on installing and setting up Squidguard on pfsense(URL blocking, useful for first level blocking of torrents etc and getting to them)
Squidguard



Info on the webpages you'll need for digging into captive portal.

On both NVu and Kompozer, this guy has step by step info pages on basically anything you can do with the packages. They are very similar to one another, so what counts for one counts for the other. Good resource for the beginner.
The Site Wizard

The stuff I'm busy working on now, liquid or Fluid webpages. These are designs/ideas to help you get your webpage to automatically fit itself to different devices. Good info here with some sample code.
Liquid Layouts

Some more info on different ways to achieve fluid design pages and the differences/pros/cons of each option
Fluid Layouts

Tips on background colors in webpages
Background colors


By no means am I saying this is all perfect links and info, but for me that started out and managed to do what I did in two weeks, this was the info that got me there, Barring a few other tips where Google became my best friend. Just keep your wording specific.

As I go along with this, bettering and learning I will keep on adding links and info for anyone who'd like to play around or build on such projects. If you ever need help, feel free to contact me directly, for any other hardware advice like switches and AP's included. If you are in Joburg, you are most welcome to come visit me at the office when I'm there and I'll gladly take you through what I'm doing/have done and if you have something to share, please do.

Wifi is close to/if not The Most Important amenity in any hotel these days and it can only ever get better if we work at it.

 

[Phence]

I can see why that internet was abused if the wifi is free. These days even parking at the hotel you have to pay for. So free wifi...ja.

That pfsense usage on that hardware seems so low that it is almost a mistake! What make is that network card or is it integrated into the HP blade server? Supposedly the cheap NIC cards can cause high CPU usage, you must have gotten decent ones?

Sent from my SM-G920F using Tapatalk

 

[Presler]

2 ports are embedded and the other two sits in an HP add-on card. All 4 Broadcom Gigabit enterprise level nic's.

The little E7300 machine that runs the Restaurant has 6 network points, all intel gigabit and the unit is a 1U rackmount purpose built for the exact use. So I think you right, proper nic's does make a huge difference in capability.

After some pondering the decision was made to go full on free with the Wifi. Firstly the hotel group marketing the hotel changed their setup from daily 100Mb free/pay after to daily 500mb free/pay after(standard throughout all hotels so this is one of those things where we had to comply). The other part of the decision came because other hotels in the Sandton area started to also give away some sort of free wifi(limited speed, some stuff blocked, etc etc) At that exact point The Maslow just up the road from us opened and in trying to compete they kept their room rate same as ours but offered free wifi. So in order to stay in competition, we thought the concept to be good. Advantages of course is lowered costs per month as 3rd party is out the door and secondly, full control in our hands. No more logging calls and waiting for this port to be opened or VIP guest MAC allowed full access, we can now make changes in real time so to speak.

Remember one thing though. While cost is always a concern to any hotel, (because at the end of the day everything costs money) the end result is not keeping the bosses happy. Everything we do is to make the visit and experience for each and every guest as simple, convenient and trouble-less as possible. It is always about the guest.