APNIC Home
APNIC home / Meetings / APNIC 24 / Program / APOPS / Transcript . .

APOPS transcript

Session one

Session two - IPv6 Operations

Session three

Session One

Wednesday 5 September 2007

1130-1300

PHILIP SMITH:

Can we all settle down, please? And those who are outside, please come back in.

So I want to introduce the first of the three APOPS sessions that are going to take place for the rest of this day here in the Crystal Ballroom.

As was explained at the start of the opening plenary, APOPS is really the Asia Pacific Operations Forum. What it does is really the main plenary session. It's the main plenary session at APRICOT and the main at APNIC meetings. And with the joint APNIC-SANOG event here in Delhi, what we have done is we've basically branded what would normally be the SANOG plenary session as the APOPS session, to indicate the joint meeting. So we have three sessions here. I'm going to chair the first one. In case you don't know, I'm Philip Smith, I'm one of the chairs of APOPS. The second session this afternoon will be chaired by my co-chair, Hideo Ishii, who is sitting at the end there. And the third session this afternoon will be chaired by Gaurab.

So we have four presentations this morning. Before I start, I'd like to ask the presenters if they could speak clearly and slowly for the benefit of the stenographers. Likewise, for you, the audience, if you have any questions you'd like to ask at the end of each presentation, please use the microphones because the session is to be recorded and is available on the Internet as well. So please state your name and affiliation before you ask the question just so we know who you are.

I'll just mention a couple of other things. The Lightning Talks session is tomorrow afternoon. If you're interested in presenting a Lightning Talk, please let myself or Gaurab know if you're interested. An email has gone out. You can respond to that if you received it. Otherwise, contact us here.

Also visit the local conference website - www.conference.sanog.org - for up to date information about that. There is the meeting noticeboard, which I'll pop briefly up on the screen as well. And you can go there for, again, up to date information about the meeting.

So, let's move on to the presentations. The first presentation we have is from Abhishek Aggarwal, talking about isolating suspicious BGP updates to detect prefix hijacks.

Isolating suspicious BGP updates to detect prefix hijacks

ABHISHEK AGGARWAL:

OK. Good morning, everyone. Hi, I'm Abhishek Aggarwal. And I shall be presenting some preliminary work in isolating BGP prefix hijacks which were done as part of my Masters thesis in Delhi, under the guidance of a professor and a doctor. The process of this is BGP updates prefix hijacks but I'll be talking about isolating suspicious updates to detect prefix hijacks. So 'isolating' is underlined to indicate the focus is on isolation. We're not trying to solve, we're not trying to solve the BGP prefix hijack problem here.

BGP prefix as a problem has been studied in both the network and the community for quite some time now. But the main - we've seen cases like the AS 912 incident which around 100,000 prefixes which led to a network outage. This was an example of accidental misconfiguration leading to a network outage. But prefix hijacks can be done deliberately for profiteering purposes, as for sending spam and automated mechanisms to isolate and classify ASP prefixes are actually needed.

Let me talk you through a basic prefix hijack example. The blue bubble indicates any network, maybe the Internet, and the white bubbles actually indicate the autonomous systems. On the left, we have AS - announces a prefix and the rest of the network start sending traffic for the prefix to AS 52. On the right, we see that AS 110, if it decides to hijack the prefix which belongs to AS 52 and announces the same prefix. For ASP, this newly announced one is highly attractive, as compared to the one from AS 52. Therefore it switches, it switches its traffic and actually starts sending traffic to AS 110. And hence as well as the rest of the network is concerned, this area, this prefix gets hijacked.

So normally what we would observe is it would get, it would see them announcing the same prefix which is AS 110 and AS 52. Normally we should have been able to figure this out. Or a BGP prefix hijack. We have valid cases on the Internet where the conflicts occur, as the next example will show. So here AS 1 is normal. AS Y and AS X are providers. Both of them announce the prefix that belongs to AS 1 to the upstream Internet. So the AS in the upstream in the Internet will see two announcements for the same prefix with different ASs. Here is a case, this is a conflict but this is not actually BGP prefix hijack but a valid case.

So our objective is to isolate suspicious BGP updates for further analysis. And objective, as I said earlier, is not to solve the prefix hijack problem. While we are not solving the prefix hijack problem but we still believe our objective is worthy because this would help the network operators to actually look at the few interesting suspicious updates. These suspicious updates can be investigated further and actual BGP prefix hijack incidents can be detected.

So the basic approach - the basic approach that we follow is we observe the state of prefix, and we analyse past BGP data to try and establish what is normal for a prefix. Once we've established the normal behaviour for a prefix, we can analyse the new incoming updates in an online fashion. And isolate them as either suspicious or classify them as normal.

To build the state information, we take the prefix and associate the state with it. Origin A is the state that we associate with a prefix and with the router, we drag the changes to the origin of the prefix. So whenever the origin of the prefix changes, we monitor certain properties, as to establish whether it is a safe change or not. So during the monitoring period, we're able to establish what would be the normal changes for the state change corresponding group of the particular prefix. We use this knowledge when we're doing the classification.

So these are the main features that we use in order to classify the state changes, safe or normal. So first one is change in percentage hold time of conflicting ASs. If we have a prefix which is being advertised by AS 1 and AS 2, and say AS 1 holds it for 70 hours and AS 2 holds it for 30 hours, then the percentage whole time would be 70% and the percentage of AS 2 would be 30%. And it would be 70% minus 30% - 40%. As the new updates come for a prefix, we bring in the change in AS path, as a second. And the third is the AS path relationship. We go further and investigate what is the exact nature of the overlap of AS paths. There are three main categories we define. Overlapping, cross and the origin. The two parts are set to cross each other. The two parts intersect at unique points but they are different - unique. And the two parts are to be distinct if they are all independent.

So just to find this - in data collect. Our data collection set-up. We apply this approach from the point of view for single autonomous system, so we use the data collected. We use the data collected from one router and we use one month data. During the monitoring period, we process a total number of 600,000 updates, all of which we found a total of 124 prefixes. So one which has more than one AS announcing them. So these were our observations with regards to the parameters which I described earlier, in this graph, on the X graph. And on the Y I have whole-time difference. Since what we can observe is since most of them are close to 200%, what this indicates is that in most of the cases, where a prefix had more than one, it was a single for most of the time. This was true for a majority of prefixes. This was our first observation.

In this figure, on the X, on the prefixes, in the figure I have plotted the percentage whole-times. So for instance, take a prefix and a bottom figure, on the Y, I have spotted this. This vertical, down graph shows the majority of this for a prefix and this was the minority. And correspondingly in the lower section when we look at the graph, we see the majority here has a smaller - a shorter A. So what this implies is that for prefixes which had more than one original announcing them, the original - a shorter path, or a more attractive part. So this establishes negative correlation between the change in percentage whole-time and the AS part of it. So what we're trying to say here is if we combine observation 1 and 2, what we claim is that whenever the state of a prefix changes, there is a new AS, announcing that prefix. If the percentage whole time is negative, then there's a majority versus a minority. The change should be positive. That's a new part which should be longer. If that's not the case and the new part is actually shorter, that is breaking the negative correlation we have established and we become suspicious. And that is exactly how a hijack is supposed to spread. So we further investigate this case and actually look at the nature of the AS path overlap. On assessing the nature of the AS path overlap, we found for almost 88% of this the conflicting ASs have overlapping paths. And the other two cases were a minimum 5% and 6%. That's the cross in the distinct path.

So what we conclude from this is that whenever during a conflict, if the two AS paths are overlapping, we go forward as normal. If they're distinct, crossing, we deem them as suspicious.

So putting it altogether, we would say a potential hijacking AS would have a negative, a low percentage whole time. A negative would have a shorter AS and a distinct is path relationship. These metrics give us during in which the first phase, the first phase is the warm-up phase where we would actually run through some part of the data and build the information to do the analysis and during the second part we would use a classification to actually analyse the data, depending on the state information we have been given. The classification is presented in the form of this. I would not discuss this part of it. I want to emphasise this branch. So, as we had when we are monitoring the updates, if there's an announcement, and it causes the change in state of a prefix, it's from a different original AS path, it causes a conflict. Then we analyse the three parameters, with the percentage whole-time change. If the percentage whole-time change is negative, that means we're shifting to a major possessive. If the AS path changes and is negative, that's a violation of the correlation and that actually means that suddenly a new original AS has popped up which has never possessed a prefix for a major time. And it's providing a more attractive path which is how we expect a hijack to spread. We go further and analyse the actual AS path overlap relationships. If they're distinct, they are highly suspicious. They are cross, overlap, then they are suspicious.

So we began the classification on one month's data. And we processed a total of around 600,000 updates and out of these the MOAS conflict were 958. And they were suspicions. 60% of them were highly suspicious which is the negative AS path. 11 of them were medium, which with the characteristics, plus the AS path relationship was overlapped. There were low 67 in which the AS path relationship was distinct.

So one interesting thing we have encountered is the high suspicion - this is a /24 prefix which was active. It's origin AS was 5050. And this origin possessed a prefix for around 99.9% of the time. A shorter AS path, as we can, and it had previously possessed a prefix for a very short time.

So this update was flagged as highly suspicious by our system and actually, according to us, needs further investigation. This, in fact, forms the basis of a further, for this study, and the future work, which will after isolation, we will try to validate these by actually conducting the prefix folders or through, or by inquiring in the data plan.

So the main conclusions from this study is to isolate suspicious BGP updates and the percentage whole-time change and the AS path change. And they are useful features to isolate suspicious BGP updates. So currently we're working on this and looking for more data to actually run it on different data sets to validate our findings. And build a community database of prefix hijacking incidents which can be treated as base truths for further studies. Thank you for your time. I shall take questions.

PHILIP SMITH:

Are there any questions?

TAIJI KIMURA:

I'm Taiji from JPNIC. Thank you for the very interesting research. I'm looking forward to seeing this. And one question is: you used the past data for detecting the prefix hijack, right. Did you have any consideration on use of the data or the IRR data?

ABHISHEK AGGARWAL:

We are trying to base our study from an autonomous system perspective. We wanted to build a system which can be deployed around this system so it can actually immediately start processing the updates that are coming inside the system. So with the IRRs, where they collect their data, does not fit in to our scheme of things.

TAIJI KIMURA:

Thank you.

MARK SEWARD:

My question is you're taking data from a network, have you done any analysis in looking at it to promotion of a live Internet? Are you looking at information feeds from the public Internet at all, down to any analysis?

ABHISHEK AGGARWAL:

We haven't been able to do it on the public Internet data. Our concern is with the prefix hijacks which would actually be available more in the Internet live data.

MARK SEWARD:

Probably quite an interesting area of security research, so I wonder if you're kind of cooperating with people in the security, on the security industry? Is there any correlation between what you're doing and what's currently - what the current state of play is there? Is there any cooperation between what you're doing with your research and the security community?

ABHISHEK AGGARWAL:

It's a hard area. I picked it up as my Masters thesis, based on a NANOG paper. The community is pretty active in this.

MARK SEWARD:

Is the code you've used publicly available?

ABHISHEK AGGARWAL:

Yeah, I can give it to you offline.

MARK SEWARD:

I think it would be useful and interesting for people to have as a tool.

ABHISHEK AGGARWAL:

I would love to share it. Thank you.

PHILIP SMITH:

Any other questions?

I have one. So data that you're looking for, does something like a live BGP feed make sense or are we looking for static BGP table?

ABHISHEK AGGARWAL:

A live BGP would be ideal but even a static BGP would be good for us. We have a system in place that we can feed it as a live feed to our systems. For the offline analysis we can do it.

PHILIP SMITH:

Sounds to me you could get a live BGP feed from one of the ISPs here in India. There will be lots of people who would probably be willing to do that, I'm sure. Places like Route Views and the rest have lots of BGP data. I'm not saying you should just do that but there are plenty of places you could get them from.

MARK SEWARD:

There's a lot of historical data, different locations, there could be a lot of additional, very interesting information from analysing that. Just to see where they have taken place, past hijacks. In the length of time it's been in effect for and then correlate that with other security events. Just a point of interest.

ABHISHEK AGGARWAL:

Thank you.

PHILIP SMITH:

Thank you very much.

APPLAUSE

Automated system administration - Devdas Bhagat

PHILIP SMITH:

Next up, we have Devdas Bhagat talking about automated system administration.

DEVDAS BHAGAT:

OK, so this is more from my experience with working at a hosting provider rather than this point of view. But the same logic still applies. Even if we're looking at managing networks, which the same thing was said in the morning, you should try and keep stuff identical and simple.

System administration mostly says we're all individuals. Every system is supposedly unique. Everything has got something different. Different IP addresses, maybe different operating systems. At the end of it, it provides the same service, just not the main server. Somebody makes a bit of change and then another change and at the end of it, all your systems become separated. They all become individual systems and somebody needs to remember how to manage them. It does not matter how much documentation you have, people will not read it.

First lot, there are unique systems. They're hard to replicate. The first system goes down, a crash, we still have a problem. You lose the data for any reason. Oops. You can't read it because it's all in somebody's head. This person gets hit by a car, "Sorry, dude, can't maintain that system because nobody has the knowledge of it."

Am I going too fast? Services end up going all over the place and at the end of the day you still can't manage them. They can be painful to manage. They're not actually all that different. What we have is a bunch of identical machines providing the same services. Some are different but most of them are the same. What we need to do is have a system that lets everything converge in together. Every change that you make brings all your systems to a more homogenous environment. It is showing one machine and every machine of that class is exactly the same. They may have different physical specs but once you get past it's still the same. They can send mail and still be the same. The same mailbox here, the same mailbox there. I don't want to worry about what is different on those.

First of the three-step solution. We have a problem. So let's have more people to fix it. The first thing that everybody does, it is getting more complex, let's hire more people. It's taking up too much time. More people is more complex. Your problem is not in the whole system environment as such but in the communication between people.

Someone says more people on the software project will always make it later. There's a fairly good analysis for why. Again, a step point, you have more complexity. A lot of companies like to do this - make it a process! I'm not sure if 'processify' is a word but it sounds good. You have the methodology in the assignment or you have other stuff. You end up with blanket checklists. You make a process, follow it religiously. If it fails, "Sorry, dude, somebody else fucked up, it's not my fault." There's a problem with this. ITIL makes it obvious. There is a centre of management database. Everything should go in to the CMDB. You have a change process for everything. A change is coming. Any change is a change request, you have a bunch of people work on this and it goes in to the CMDB. "I don't know how to do this, let's put it in the CMDB." At the end of the day, nobody knows what's going on in the CMDB. Your complexity is that it's sitting in the CMDB. There's still no documentation, no clarity. Nobody knows.

"Go away or I will replace you", is the standard small shell script. You take their jobs, replace them with a small problem and, "Now you can go off and do better stuff." It should be scripted. Follow the methodology. If you need to do it twice, you make it a program. You make it a small program and you keep extending it bit by bit as you need. You can end up with a 35,000 line transcript, which we actually have at this time in work. A lot of people do this because most of the assignments go, "I'm not a programmer." How do you point and click. They actually let you do that. But it's not very commonly known. If you're going to go to the programmer side of things, you write most of your stuff in the memory management. You don't worry about how to combine stuff. It gets repeated identically all the time. You don't just apply them to our side.

Benefits of automating stuff - auditability. It makes work consistent. Those are major benefits. How often have you walked in to work looking at a problem and go, "Oh, shit, someone else made this minor typo last night and we've been losing out on stuff and have been rejecting mail for the past 12 hours." Or somebody made a change on Friday night and you walk in Monday morning, and then for the past 48 hours there has been a screw-up.

More benefits - make work testable. You can give guidance, make it work in a virtual environment. And we know your changes will not damage anything. You can make sure that, "OK, I can work out the software." When you're servicing 50,000 users on a single box, then you can have 100 calls to your helpdesk.

You can actually have a process that says, "This has to go on to the test infrastructure. It must be documented and automatically it is done." It needs to be done before it gets deployed in to production. Make it be faster and bigger. For the end of this year, my goal is that my entire department will have full assignment work as such - 15 people. Maybe another few helpdesk calls. No more work. That's it. We need tools. Currently the state of tools is not very good at the moment. No-one does it except for John McCormack. How many people do you see offended - one. I sympathise.

OK, this is what assignments do, this is how we need to make it work in 100 boxes. Let's write down all the steps we need to do. There are better ways to approach the problem.

But the rest of them are all aimed at basically replacing this.

The next few slides. I'm more familiar with the tool, so I use Puppet configs. You have a whole bunch of people who can make the test, that's the only way you can test it. It will take about a year's worth of testing before you're willing to deploy.

OK, let me jump through this. This is my example config. Deploys postfix on the box. I need to configure it like this. I spent 16 minutes writing the definition for the first. The second one was a copy and paste. The third one, same amount of time. Deployment time for this has gone down from two hours to five minutes. If I need to deploy 10 boxes, it's still the same five minutes because most of the five minutes is penned in. We need to do this, log in to the box. Do the change. Close it. That's about it. It takes me more time to do that kind of work than to do this. If you want to change the configuration, I can do this. I just have to say, with the allocation thing, a blacklist file. A blacklist file there. A refresher, actually the postmap, that's a postfix specific thing. Basically you're making maps. And I can automatically get it deployed out to all my boxes. Puppet does that for you. I no longer have to worry about, "Have I forgotten that box?" It will be in the same state as any of the box, they're all identical using Puppet.

Alternatively, I could define my boxes like this. We use this class for anything else. So if I have a website which needs MDB, Puppet will start off saying that I don't have that but then Puppet will install it all for me, the configuration. I'm not interested in knowing how it all works. I don't care if it's Solaris or RedEye. I don't want to know all that. I want it all in the box.

This works, migrating between operating systems between hosts. You need to express your policy. Normalise it. Define it once and then you simply make a reference to it everywhere else. There's a lot of mathematical theory that goes in to building these databases. Building these databases is like building a relationship database for your system conflict.

Benefits - you don't need that many administrators. Yes, I was told I should not put this thing in because a lot of people say that we don't need all these. Describe it - it's no longer the assignment's problem of how they work. For most of the time when things go well, you're no longer worried about the final days of how they're implemented or differences between systems like that. You say, "OK, I want this on the box." How do you define it - Puppet will take care of it for you. Puppet will do the script. It's no longer a problem for this assignment. This is a big change. What happens when you have the box and they're extended down there? If you miss out on one change, it's fine. You can push it out to the next change. Two changes, three changes, four changes and then you realise that you have another box that has been down for two changes. You cannot maintain client state here. Don't complicate it. Let them figure out the stuff. What is its current state? What the state should be and synchronise the two.

Recommendations - Google does it. They have their own stuff that they don't publish, which kind of sucks. One company manages it with lots. Servers are located so they have not monkeys running around if need be but there are four. If they're afraid somebody is going on vacation or whatever, four people. Not all things can or should be automated but quite a bit of it can. The decision-making process, you shouldn't but with implementation, you should.

Not all the tools are equal. They have their pros and cons. The tools are starting to offer things but there are completely different views of the box at this point in time.

There are great reporting but it doesn't have the same detailed internals that Puppet does. Puppet has fairly low reporting. It can dump but there's not much more of reporting other than that. There's reporting in that but it's building. People don't like changing the way they work. This is fairly simple. You spend four hours, six hours building a config. And you start deploying the boxes, have your untrained guy walk in in 10 minutes. No training required or anything required. This is a great change to your management policy. People are used to logging in to the boxes and working. What you need to do is stick your conflicts in. After that everything else is automated. You just check your conflicts in to control and then check out your conflict, it does them for you and it's OK.

No problems. You need few administrators but you need them with a far higher skill set. They have to understand what they're doing. You can have a bunch of more monkeys doing all the grand work if you need to but in general you're replacing them with software. So you need fewer people, four people is more than enough. But all of them need to be correct at their job. They need to be able to decide and implement policy as I'm going X, Y and Z. This is what I want on these boxes. This is what the system will require and this is what needs to be in the boxes. This is what not needs to be in the box. It takes time and it requires management buy in. That's a problem because you still need that infrastructure. Hopefully everybody has one but a lot of smaller companies don't. It's hard to get it because most of them will prefer to say, "It's easier for me to get cheap, unskilled labour, rather than trying to get somebody who is skilled enough to use this tool." If you have the standard kind of idea, "I need someone with X years of experience." So that's a standard joke where you have, "I need somebody with seven years of experience on Windows 2000 or someone like that." OK. Questions?

MARK SEWARD:

That was a fantastic presentation. A lot of this material is very useful and needs to be, I think, communicated to the larger ISP audience in particular. Are there any forums or perhaps central locations where best common practices are documented? So this sort of approach you're talking about here is...

DEVDAS BHAGAT:

Infrastructures are about website. There is a website, League of Professional System Administrators. There is an automated config management system. There is a config management mailing list as well. But at the moment that's pretty non-busy at this point. I've not seen any traffic on that list for the past few months.

MARK SEWARD:

How do I do this. How do I approach this and take the software that is out there? You've shown there are many packages and a lot of people have time in developing these, but how do I as a small service provider go about automating all my processes and procedures? Most people, I would think, probably do it their own way. They put their procedures together in their own way. Is there any definitive kind of reference? Are there books that are published that take you through?

DEVDAS BHAGAT:

There are a couple of books but they're old. There are no current edition of best practices. Infrastructures is probably the best quick reference to get started off. They have this paper that they have published and they have a process that they follow there. This was published in 1994. Still speaking about the fact we need to automate stuff today. I'm not sure how well the idea has been communicated around. But, hey...

MARK SEWARD:

I think the opportunity exists for material about this.

DEVDAS BHAGAT:

There is a future. If you're looking at infrastructures, with hash infrastructures and Puppet. There is probably others for all other products as well.

MARK SEWARD:

How long has the Puppet product been around?

DEVDAS BHAGAT:

For about three years now. 3.5 years.

MARK SEWARD:

Fairly widely deployed?

DEVDAS BHAGAT:

No, but it is growing. Probably one of the more interesting tools. At least according to what I'm hearing on LOPSA. There are people looking at it and deploying it, rather than just figuring out what to do with it. The Puppet website has a whole bunch of recipes so you can go there, copy, paste those down and do your work.

MARK SEWARD. There are references there?

DEVDAS BHAGAT:

There is.

MARK SEWARD:

That's the point of presentations like this, I guess. PHILIP SMITH: Anybody else? No other questions? OK. Thank you very much, Devdas.

Experience in wide deployment of wired/wireless/fiber in Kathmandu valley - Samit Jana

Our next speaker is Samit Jana from WorldLink.

SAMIT JANA:

Good afternoon. I'm Samit from WorldLink Communications. My presentation is a little bit different to others. I'm not too technical. I'm not doing dos and don'ts or anything in technologies. I'm presenting our own experience, how we were able to scale the layered network. We're not using QTQ, just a simple layer of the Internet extending over the metropolitan area of Kathmandu valley.

So this is the outline of my session. The challenges we face in Nepal to deploy broadband. Number one, we have a political instability, which makes it difficult to make a long-term investment and introduce new technologies.

We're still connected to the Internet backbone via VSAT, expensive and very slow.

Similarly, many people still don't know what is Internet, lack of education and awareness. Similarly, the GDP of the people is very low.

And our landscape is also very difficult and we have plains, hills, as you realise, mountains, so to cover a large area also is a very challenging part.

But still we have managed to introduce broadband and we were able to provide the cost effective broadband solutions. So our vision mainly focuses on the need of the people. That is to hook them up into the Internet and now, so that our main vision is connect everyone in Nepal to the Internet at an affordable rate.

So those who don't know what is Kathmandu, it's the capital city of Nepal. It's a very small city, 280 square miles. The population is about 2.8 million in the valley and Internet users are probably around 3 million around the whole country. And according to the ISP Association of Nepal, we have 33 operational ISPs currently operating and we have 60,000 active user accounts in all ISPs.

As I explained earlier, we are still connected to the VSAT. They have acquired fibre from VSNL but not of the ISPs have produced any transit yes. So this is how Kathmandu looks. It's very small.

The diameter from here to here is approximately 60 km so it's pretty small.

So WorldLink, we started in 1995 with a single VCN holding but currently now we manage 102 servers and we subscribe over 65 mbps of satellite bandwidth. We have over 25,500 active accounts and 300 employees. We have Internet market share of 41%. Network coverage all over the valley as well as all the major cities within Nepal. Currently we have /18 IPv4 address space. We also collocate a service and we use unique applications.

So this is how our network looks like. So most of the places we are still connected via the VSAT and these blue lines, are the lines and recently we managed to put in a wireless link but many of the places we couldn't see are the red lines. These are all back-up links which can span up to 100 km between one hub.

So typical ISP set-up, nothing so special. We are connected via the VSAT up in to the Internet in Hawaii. We have a collocated server at Hawaii, where we have a web and DNS server to serve the international customers. We have a bandwidth managers as well as a bandwidth compression at the broad end. We use a Linux manager for the QoS and output setting and this compressor is the compressor where we hack the GBS source code and I think maybe within three or so months we'll make that open source for the use of everyone. It saves 10% of satellite bandwidth. That means we are subscribing 65, we are processing more than 70, 72, something like that.

So we have one switch running a bunch of servers to manage the route balancing for the services like SMP, PoP, DNS, web, etc. Here - this is our upstream and we peer within an exchange from here. We have routers which connects to PoPs outside the valley via VSAT or via the wirelessly. Similarly, we have a knock-out here. Even the knock is behind the firewall, we have access to all the network in case of any failure. We have all the servers and bandwidth managers.

So why do we need the wireless broadband? Because, until 2002, we are fully dependent on telco leased line and the point-to-point wireless connectivity and, as we all know, it is very expensive and cannot scale much, so we thought that point to multipoint connectivity would be a scaleable solution.

So in late December 2002 we introduced a Motorola Canopy point-to-multipoint platform for the first time in Nepal. We started with static IP allocations to all the customers and by the end of 2003, 400 new medium-sized corporate customers subscribe for the Internet and the VPN data service. So as soon as we introduced the wireless broadband, the bandwidth licence has jumped from 8mbps to 18mbps in one year and the price went down to $10 per month.

Oops, what have we done? We have leaked the private LAN in the public network. So our network was down a couple of times due to all these things. I don't think I need to mention all these things because it's natural, if you extend the LAN, you will all get this. And there are a few related problems, which stopped us to scale the network. At this point in time, we thought that we would stop doing it. We thought that this might be impossible, but things changed.

As soon as the Motorola, they introduced the new firmware with the packet filtering and the NAT in the subscriber of the wireless, it stopped many of our problems, but still we made a 24/7 proactive wireless monitoring team within our network and it runs packet sniffer and filtering stuff and we employ the broadband routers to connect directly on the PC of the customers. We ran scripts to reboot it automatically if there is any GBS errors and we automated a lot of things. And we got encouraged that Layer 2 can be scaled.

In late 2003, we did a PPPoE-based service. Gave us traditional dialogue. We introduced hourly volume-based broadband connectivity. The service was very successful. We have over 700 customers within one year. Again, the price went down, around $90 for 60 kbps per month. Currently we have 67 access points servicing the Kathmandu Valley.

So this is how our connectors look like. We have six access points covering 360 degrees. We have 25 megahertz of separations and some clusters. We even have two access points. This is what our tower looks like.

This is subscriber module. This is taken from my house.

If the customer is a VPN subscriber, we only allow the VPN connection. So if the customer is using a static IP address, then we allow all IP and all others.

So, there are not so many major problems in this space, only a few bottlenecks. We added additional access points, upgraded the classic AP to a newer one and we even segmented the PPA customer and the static wireless customers and then we ran out of the /19 IP address and we acquired another one. We are confident that layer 2 can be stable.

So why we need a Last Mile Ethernet? Because we need to migrate the dial-up customers to broadband. So most of the dial-up customers, they are paying more to telcos than to ISPs. They are paying $10 to $20 for the Internet as well as paying $12 to $25 to the telco using a home line. So we cannot migrate those customers, the end-users, home-users, for the expensive broadband because the subscriber module is still very expensive, it costs around $300, $400 for one module. So we need a last-mile solution. The solution is simple Ethernet. Sounds stupid, but it worked.

So, in the first place, what we did is on the electricity poles, we designed our own metal boxes, waterproof metal boxes and in the metal boxes we mounted an eight-port VLAN mobile switch and the switch costs less than $20. The poles, it was hardly 100 metres apart and all the switches are cascaded linearly. And this way, wherever we have wireless connectivity, we can put in Last Mile Ethernet.

So this is how it looked like. Nothing so special. It's connected either by fibre optic connectivity or by wireless. We have a remote switch out here. If the switch hangs, we can power recycle the switches. And we have a custom of cables bundled with electrical wire to power supply all the switches.

So we have Ethernet segments up to 20 cascaded switches covering around two kilometres. So all the ports can reach - one Internet segment cannot exceed more than 100 metres but we are operating up to 150 metres because we are running - every customer was kept in VLAN. PPPoE connection is being allowed. By the end of 2006, we have over 4200 customers being served by the Ethernet. Until now, we have only deployed 600 kilometres of Ethernet cable and 6,500 VLAN switches. So this is how the VLAN on the last mile Ethernet VLAN is configured, so every port on a separate VLAN, the sport number is configured as multi-VLAN, member of all the VLAN. In other switches, the port goes to VLAN. This part either goes to another switch or to a subscriber module or to the catalyst switch port.

So the major problem we faced in this is switch-related, because we are using very low-grade switches, so we can migrate to the high-end switches, because that usually increased the total costs of the users. And sometimes the servers go down with the customer's PC so we start the to use Internet service protectors. Sometimes the switch hangs so we install remote-accessible power control to recycle the power switch. Can take the switch and reboot it automatically. Replace the power module with 80 to 100 A/C. When you go on extending the power, the voltage gradually drops, so our first version of the power cable, it has a low grade of power so we increased the size of the copper wire and we managed to change the power supply. It can operate in low and high end.

Occasional Internet cable switch breaks. And this one is very interesting. Faulty Internet cables and broadband routers of the customers creating a loop in the network and it creates like a broadcast slump and it brought down our network a couple of times into a complete hump. This is something like, if you take a switch, a cross cable and connect the two ports, so what will happen is the traffic will loop and nobody will be able to connect. The problem is not in the metro Internet because the catalyst has a loop mechanism which can disable the port as soon as it sees the loop pack, so it will put the port in the error disabled state so that segment is off. But the problem is in the wireless, because we don't have that feature in the wireless, so the problem still exists in our wireless network, but we have a mechanism to detect that and isolate the segment very fast.

So as the number of clients increased, the subscriber modules were incapable of handling the load so we migrated to the optical fibre network so we gradually start to replace the subscriber modules with the fibreoptic. We have already deployed 450 kilometres of fibreoptic cable.

So this is the metal box mounted in the electric pole. This is the inside of the box. You can see the switch, the link and also see the wooden box to protect from electrical surges coming up from the electricity pole and this is the power distribution switch. You can see this customised cable. Here we have a cable and here we have the power wire.

So these are the features which we use. Not so common. We use RSTP, we use spanning tree. We use storm control and MAC security, we use protected ports and we also use port-based MAC and IP LAN. We use all these.

So this is how our network looks like. No so complex. We have two different rings, for a wireless and for an optical fibre. You can see the wireless extended over the SN up to the end-users. This runs RSPT, we have a failover time of less than five milliseconds and we have - since all switches are manual switches to find out whether this segment is alive or not, we have an IP switch at the end. We even have a hot spot running from the same network so if any customer requires a static IP, then they need to have their own CP at their end and we add them to turn it into a switch. So we have a FreeBSD service and we have a Linux bandwidth manager on the edge.

So the main challenging part is monitoring the switch network. So how do we do that. With the open NMS and the Nagios tools, we developed our own application to trap the active MAC address and analyse the packets in the network. Did a report on the aggregation switch which trapped all the packets in one server. Actually, I'll go to the next slide, which will be a little bit clearer. We have a packet analyser out here. This is our aggregation switch. This is a mirror board, which sends all the packets of this whole network out here, so it traps all the packet information. It puts it - it stores it in another database for the historical data and on the fly it sends it to our monitoring servers. So this monitoring server, it can, on the fly see whether this user is alive or not, it is connected on which board and what traffic it's sending, how much packets it's sending, everything, and whether that user is online or not, it makes also to the RADIUS server as well. So if anything changes in the database, it can directly send the changes into the memory of this monitoring server.

So this is how our monitoring looks like. This is all our PoPs. Green means it's active, red means it's inactive for some period of time. Grey means some activity is going on. So if you click any of these sections, then you'll get this type of topology. So based on all the information, it can automatically create a network topology. So these greens mean the switches are alive and the black means it is dead for hundreds - from last 100 seconds, that means it's not getting any packets from this network. So every one second, this segment changes the colour, so eventually it runs into the black colour.

So if you click on any of these sections, it will pop this type of applet and you can see in which port, which user is connected. So green means this user is alive. So if you click on this port, you get - you can get all the user information, his MAC address, his download speed, his IP address, and if you want an on-the-fly, graph, could get that. We are still optimising this. We can even - we can graph anything, we can grab it from the server. So that's not a big problem but if the user is disconnected and if the customer is alive, it will dynamically create the graph on the fly.

So this has helped us quite a lot to monitor our online wireless network.

So, currently we are Googling for this additional features on the subscriber model. We have already sent a request to add these features. So if they add these features on their future releases, then it's fine. Otherwise, I think we need also hack that or do a reverse engineering. With the metro Ethernet, we plan to replace all the unmanaged switches with managed switches.

So our future plan - get the fibre backbone as soon as possible, get our own terrestrial wireless connectivity and we are planning our broadband power. That's the end of my session. Thank you. Any questions?

MARK SEWARD:

Mark again. You're saying you use a custom-modified FreeBSD to enhance your natural connectivity. What does that change in terms of latency? You've already got a latent -

SAMIT JANA:

I'm not sure because I'm not a programmer, but in our test it's less than one millisecond.

MARK SEWARD:

Oh, really? That's very quick. No large changes in the amount of buffering you have to do? It's pretty quick?

SAMIT JANA:

Yes.

MARK SEWARD:

OK. Second question for you. I notice all the metro Ethernet you use brings it back where you've got PPPoE servers everything. Is the amount of peer-to-peer travelling on your network an issue? Does it mean that you have customers out of their location, wherever, at the home, talking to other homes in the valley, do they all have to traverse through your network back to the core, to the PPPoE server and transfer there and come out again? Does that cause capacity issues for you?

SAMIT JANA:

Not yet. Because the traffic is currently only the web browsing and e-mail. So a lot of people there are not using any VOD or any live streaming. It's basically only the Internet and e-mail traffic. Capacity is not an issue right now.

MARK SEWARD:

People aren't bittorrenting in Kathmandu?

SAMIT JANA:

Most of it is web and e-mail. We have a gigabit - capacity has not been an issue. We have a capacity of 95mbps of traffic. So that's not an issue for bandwidth capacity.

MARK SEWARD:

You're running VLAN spanning tree for redundancy. Do you have any situations where the fail over time becomes a problem? Are you saying it's failing over in five milliseconds. Is that regular? Is that normal?

SAMIT JANA:

As the number of switches grows, the recovery time will gradually increase. We have currently 14, 15 switches and we're getting a recovery time of four or five milliseconds. Maybe if you add more switches, the recovery time can go higher. At that time, we can use NSD to segment a section and to get the better recovery time.

MARK SEWARD:

What percentage of your switches are still old, unmanaged boxes, versus managed 3,500s, or whatever you've got?

SAMIT JANA:

Almost all.

MARK SEWARD:

They're all unmanaged?

SAMIT JANA:

6,500.

MARK SEWARD:

That's a lot of switches. Thank you, that was a very good presentation. Thank you.

JONNY MARTIN:

It's not so much a question. I'm a big fan of pragmatic networking like this. I've done some similar stuff back in New Zealand, but this is 100 times bigger than what we've done, so it's really good to see.

Secondly, what is the alternatives that are offered, so the incumbent telcos, DSL or wireless or some similar sort of product? And how does that fit in with what you're doing?

Basically, what does the incumbent provider offer?

SAMIT JANA:

OK. You mean to say that - basically, our main competitors, they are also using the same - they are also using the same wireless which we are using, but they are not using the metal Internet type that we are deploying. They are deploying television and Internet on the same cable, so that was one alternative. And we don't have any additional connectivity and they are soon introducing connectivity after one or two months. Besides that, I don't think there are other alternatives. Maybe some are providing a wireless service as well, but not much.

JONNY MARTIN:

OK, great, so the sorts of technologies you're using are accepted as the way it's done in Kathmandu.

SAMIT JANA:

Yes.

JONNY MARTIN:

Cheap and nasty. That's good to hear.

SAMIT JANA:

We need to focus on the need of the people. We need to make them aware of the use of the Internet. So once you create a market, then the technology will automatically come.

JONNY MARTIN:

That's great. Thanks.

GAURAB RAJ UPADHAYA:

Gaurab from SANOG. I should know about this because I'm one of the customers on your network. I was wondering what kind of Layer 3 protocols you use and how fast your routes can coordinate broadband. I think it's pretty fast all the time but I would be interested to know what Layer 3 you run on top of this and how it's scaling right now.

SAMIT JANA:

We use a typical OS sphere and BGP. In our layered network.

GAURAB RAJ UPADHAYA:

And for the end-users? You don't do any others, backhall back to the core routers? Even with PoPs, all the way, static routes and VLANs?

SAMIT JANA:

Yeah.

GAURAB RAJ UPADHAYA:

Works well?

SAMIT JANA:

Yes. It was all backhall to our people, on the PPS servers.

GAURAB RAJ UPADHAYA:

That's good, thank you.

PHILIP SMITH:

I think in the interests of time, we probably should move onwards. Very quick.

SHENGYONG DING:

I'm Shengyong Ding from China Telecom. I want to know if this is very profitable in your area, in your city? I know in China Telecom, that the broadband is not very profitable now, but I want to know how profitable it is in your country? Is your service very profitable?

SAMIT JANA:

Yes, of course.

LAUGHTER

SHENGYONG DING:

Thank you.

PHILIP SMITH:

OK, thank you very much for your questions and to Samit. Thank you.

APPLAUSE

Trans-oceanic systems - check list - Barry Greene

So final presentation is from Barry. It's on trans-oceanic systems. While he's getting set up, just a request from the folks on the network and infrastructure here. If you see a plug plugged into the wall, do not unplug it, OK? Let me be very clear about that. A plug was unplugged at the back and a fair amount of infrastructure was switched off. If you need to plug your laptop in to something, there are lots of empty seats here and lots of power boards for you to plug your laptop into. Hopefully we will not have to make this announcement again.

BARRY GREENE:

Move up front.

My name's Barry Greene. I'm from Cisco Systems. Some of you know me because I've been out in the Asia Pacific for quite a while. Some of you don't know me. Usually you see me as a security guy. I've been doing that for six or seven years. Before that, when I was living in Asia Pacific, one of the things we did was build backbones across oceans.

Me coming back out looking out here, one of the things I'm concerned about is - and I'm just going to make this brief because I'm between you and lunch - is the people are not realising that, you know, your link across the ocean, satellite links, terrestrial links, are things you need to engineer, and give an example - changes in, like, in big providers, US, Europe, things like that, people are moving beyond because what it used to be, was just Internet back and forth with two peering routers and people were going back and saying, "I want video and voice and lots of different things." Your interconnects are a current component of your business. It's about business. As Vijay was talking about earlier in his keynote talk, you've got to think about your business as you go through.

To give you an illustration of why I'm concerned, I'm seeing a lot of different providers as the market increases out here. What they do is they don't realise the dynamics of an engineering principle. Engineering principle is, technically, as you get a bunch of stuff coming across your backbone, you have choke points and if you want full control over your business and your networks, you've got to be able to have places where you can police the traffic and control it and have visibility. A DOSVO out of the cable community has a lot of functionality that you put inside the house. The reason you put it in the house in the broadband, whatever you want to call it, is because there's a lot of stuff going on inside the house.

You've got kids doing games, people doing peer-to-peer, people doing video on demand, the voice services, all these sort of things. To do that effectively and not give your Helpdesk, you know, constantly in call by your customers, you've got to be able to control both sides of the link. So in industry terms, it's shown that this principle is valid, you've got to be able to have full control over both sides of the link. You see that and the standards work.

On a trans-oceanic system, users of ISP have a bunch of stuff coming down the pipe and if all you do is have a circuit out to a service provider out there, what do you do? I've run into four different providers here and they're describing congestion slots and BGP is getting knocked out and it sounds like you're overfull on the buffers. Do you control your upstream router? No. So the service part is not very helpful, right. So these sort of scenarios is fine if you're an enterprise customer but you're running your business over this so you need to be able - just like you do things on your side of the link, you need to be able to do things on the other side of the link, because this costs you money. Right? That's kind of the core principle.

Think about it from a very simple standpoint. Go to your bosses, your CPOs and say, "If all we've got is a link across the ocean, really we have no control over our business because lots of stuff is coming down the pipe and we're paying for it. We need positive control. QoS, we need to control both sides of the links." You don't want to go through and, as you send packets up, you have packets come down, you pay for the packets because they're coming across your links and then what happens? Somebody drops a packet. You have lots of spam coming through. What happens? You drop the packet. You don't have any business control over that. If you want to do QoS for a customer, you can't do it on one side of a link, you have to control both sides of the link.

So this is one of the things that's a concern because if you want to have positive business control, it's something that Philip and I, when we do the workshops - and Philip says he's still doing it in the workshops - you've got to be able to control both sides of the link, engineer it as a system and what I mean by that is there's a whole bunch of stuff that you saw in the last presentation, there's a lot of things happen in the service provider in-country.

Correspondingly, there's supposed to be something on the other side of the link too, engineered as a backbone system going back and forth. The question that was just brought up by a colleague of me here about what happens when you get peer to peer between local PoPs, it's the same sort of principle, you need to be able to engineer through that. Think of it as both sides of the ocean. So my critical advice - and I'm going to give a suggestion of what we can do about this is don't be misled by service providers who say they're your peers. A lot of service providers say, "Buy a circuit off me. I'll give you a good price." To them you are a customer. You're somebody they can make money off of. They don't really, you know, care about your business success. All they care about as long as they get a pay collect and you keep on - you keep on calling them up to upgrade the circuit, right?

Sales engineers, from the service provider. "Come in. I know all about this stuff." But they're sales engineers. Their job is to sell stuff. Beware. If you're smart, any sort of sales engineer when they come in, you use it as data points. You don't copy exactly what they do, you use it as data points. You can collect data points.

The same thing from vendor engineers. They're out there to sell equipment. They may not understand your business. But sales engineers from a service provider or vendor are good points of collecting data and then really watch out because, over the years, I see this over and over and over again, watch out for the consultants from aid organisations. "I'm a consultant from country X, I'm here to help you. Please listen to my advice. I've never run a network in my life but I'm a consultant." Right? You know, over and over, different service providers and businesses who've been messed up because they listened to the consultants, right? So watch out for that.

And beware anybody who wants to have you dependent on them. Alright? Like, for instance, " Hey, connect to me and I'll allocate IP addresses to you from my IP address block." Right? What you're doing is setting your business up to be dependent on them you're taking prefixes from an upstream provider, building your network out of it and guess what? You're stuck. Right?

This is why we've got APNIC. That's why all the policies and everything going on with APNIC that's involved over the years is so each service provider as they run a business can actually have control over their business and not be dependent on any upstreams and you have choices and when you decide to go from one connection to another connection and one backbone to another backbone, you have that choice and you have that control over your business.

So the advice here is: Do your own engineering. I mean this segues into Vijay's talk, you can't let people outside do your engineering for you, you've got to do your own engineering, you've got to empower yourself and the advantage of forums like this group, APOPS, is that you can tap into your peers from all over the place, right? The advantage of having a mailing list on APOPS is to actually ask for help and find out what to do with this or that engineering problem.

So one thing that I started to do was I started brain-dumping because I was thinking as I ran into - earlier this year, I ran into a couple of service providers and they wanted to do these - how do I build my link across the ocean and they were going to buy a port from somebody in the US and I said, "Wait a minute," and I started brain dumping. Realising coming here and the power of the Wiki, is we can get a massive brain updumping going on, so my suggestion to the APOPS coordinators is to use the APOPS alias and getting people to brain dump, write down principles, techniques, tricks of how to run a successful business when you've got high latency links, engineer across high-latency links, operations over those high-latency links and we can brain dump it.

And then, since we don't have an APOPS Wiki right now, I've got a public Wiki out there we can use and we can brain dump. I created an area on it and we can brain dump in there, the knowledge from different people. There's different approaches. If you're - one easy thing to classify is there's particular approaches of work when you build and a one-gig and there are particular approaches above one gig because you have different traffic profiles and engineering things that happen with circuits and buffering and what you can do and what things you can not do with it so there's a lot of different knowledge that we can brain dump and tap into. That way, when you need to build these things out, you know, and we've got new engineers coming in and everybody always has the issue with new engineers coming in to your organisation - how, you know, how did they come up to speed? Right.

And we also have service providers out there with the knowledge set and can tap into that where you can tap into other people's, you know, hard-knock experiences. They can say, "I tried this and it didn't work. I do this now." Share that out.

That's all I have to say. This is actually like the executive summary of the slide presentation I'll put up. There's a bunch of other slides that talks about principles to kick off the dialogue but I don't want to get between us and lunch now.

Questions? Thoughts?

PHILIP SMITH:

No questions for Barry?

CHE-HOO CHENG:

I want to make a comment regarding what you say about trans-oceanic network. I think there are practical issues for Asian ISPs to set up, let's say, a PoP in the US, because people cannot easily manage router in the US and they have issues buying equipment over there, they have issues setting up and maintaining routers over there. So on the other side of the - I think you have to think about how you can set up and manage the router on the other side of the ocean. So that's what I wanted to say.

BARRY GREENE:

Good point and every service provider that I particularly work with to get trans-oceanic system up, outside their country, Hong Kong, Singapore, US, Europe, it's a matter of learning the techniques of how to do that. To get started, there are service providers in the United States, in Europe, in Hong Kong, who will lease you routers. You know, all the vendors, right, I'm a vendor and in our competitors, colleague vendors out there will set up sales contracts where you can buy in your country and they will deliver it onsite at the co-lo and there are agreements around that. Things that we've had problems in the past and - it's a matter of empowerment, teaching people how to work it. It's not a huge hurdle.

CHE-HOO CHENG:

I want to make a comment on this also. I think when you set up router over the other side of the ocean, you shouldn't rely on your ISP to lease you the router, because actually you're stuck with your ISP also with that kind of configuration. You probably need to have, need to find an independent equipment re-seller or integrated, system integrator to help you to do that and then you have, you know, more flexibility to connect with different ISPs and to switch over things like that.

BARRY GREENE:

Good point. Document the point like that. And then point to people - how do you find people to help out. Speaking as someone who stands at a microphone helping out, things that would be worthwhile for APOPS, my opinion, to get written down, right? So the knowledge is not locked up in people's heads, people have access to that.

BILL WOODCOCK:

There are a lot of other Internet service providers that are at your scale, whatever scale you're at, globally, and so, if you're sort of at that stage of trying to go from being a small multi-national or a national ISP to a small global ISP, there are a lot of other ISPs who are not in direct competition with you who are based in other countries and other parts of the world, but who have exactly the same issues that you do, right?

I mean, you may be based in India, they may be based in South Africa or in Brazil, but you're both trying to get equipment into London or New York or something like that and so what I've seen is a lot of really good support between ISPs that are of comparable size that are dealing with exactly the same issues but who aren't in competition with each other because they're based in different places and obviously, you know, there are a lot of small ISPs that are perfectly competent that are based in New York or London or Hong Kong who would love to be able to make a little bit of extra money by helping you, you know, do remote hands on your equipment and, you know, they'd love to get the experience of working on a little bit bigger equipment, a little bit larger network and dealing with engineers who are doing global-scale routing and, you know, they may be willing to get up in the middle of the night and go work on your equipment for you for almost nothing in exchange for that experience.

GAURAB RAJ UPADHAYA:

I'd like to add I know a couple of ISPs that do that and I think best example is in South Africa. They started as a small ISP, they put a PoP in London and the next thing we knew, they put out their own PoP in New York, because they are paying their telecom for the STM circuit and they were depending on a telecom to get up and running and now they're expanding into Hong Kong because they realised the benefit of running from South Africa up to London and as well as eastwards to Hong Kong and they seem to have scaled pretty well. I don't think they have staff in New York or London or Hong Kong and from the three people working for the entire network ops group in Johannesburg. So it seems to work pretty well. And if you are going to put a Wiki, then the Kiwis, who use this thing to put the servers across the ocean in the US, because the telecom won't play ball with them, so put all the content that telecom users get out in San Francisco locally and get it cheaper out for them, of course, if crosses the ocean and, you know, probably gives the telecom a bit of a hard time, where some services are slow and some certain services are fast from different ISPs or different places.

Actually, one of your former speakers, Samit, I'm one of his customers and I know when they put the servers across the satellite in Hawaii, they saved our satellite at least four to five megs of spam, this is globally. They realised that when they put in the spam filters on the other side of the satellite link, they dropped five to six megs of spam and junk on the satellite. It's a very valid point. Thanks, Barry, for bringing this up.

BARRY GREENE:

Right. And you engineer it as a system and you've got these characteristics of good points. I don't see an engineering principle over high latency links talked about a lot because there is a saving of, like, you know, where you can see bandwidth save right now, like in a spam issue but there's also, if you put servers that talk to each other on either side of the link, you get TCP window sizes increase. Instead of having your window size way low, it's cranked up because you're able to have a little bit more effective sort of dual proxy sort of thing on with it. So anyways, if you're interested in it, we'll start up a conversation on APOPS. It's something I'm going to be writing about because I've got customers out there. And just give awe scenario, this isn't just for the ISPs.

What got me shocked about this isn't so much with the smaller and medium ISPs, it was actually large ISPs that got me shocked and appalled about this. In one service provider in one country in Asia Pacific - and this is, you know, the number two provider in the country - they didn't have their own link across the ocean for various reasons. They decided they wanted to do that. At first, their design was they were going to buy a circuit. Luckily we got some people in there and said, "Actually, you get no control over that. Here's suggestions."

And they designed a system across the ocean and then we introduced them to different buddies, right, and the nice thing about APOPS is APOPS is plugged into things like NANOG and others. You've got buddies. This guy sitting next to me, Philip Smith, you put a router across the ocean and if you want to find people you can peer with, like Bill points out, there are small ISPs who would be welcome to peer with you if you drop into an exchange point or co-lo.

How to find them? Go to your buddy Philip here, go to NANOG or RIPE, and Philip will introduce you to those. Go to NANOG and hook up with someone like Vince Fuller, and they come out of that two-day meeting with 16 peering agreements. Done. The circuit was up for two days, go to NANOG and get 16 peering agreements besides your transit service on it and that's the power of groups like APOPS, NANOG and talking to peers and things like that. Anyways, we'll move this over to APOPS. I'm around during lunch and thank you.

PHILIP SMITH:

Thank you very much, Barry.

APPLAUSE

OK, before the presenters go and before everybody goes for lunch, we've got some gifts to give to the presenters. I was hoping that somebody from the ISP Association would be here but I don't see anybody to hand over the gifts, so Hideo, if you could - for the four presenters.

Sorry to run into lunch a little bit like this. We've got an hour for lunch. Lunch is out on the terrace patio so you come out of the room and turn right.

This afternoon, we've got the continuation of the APOPS session. So we've got 90 minutes for three ISPs who are going to talk about their IPv6 deployment experiences. So this is a real operational issue that these ISPs have faced.

That session will be chaired by my colleague here, Ishii. So please come back here at 2:30 sharp.

In parallel with that, we also have our APNIC Fees working group meeting, meeting in Regal 1 and 2. From memory that's downstairs when you go back into the hotel.

So those are your choices for this afternoon.

Please enjoy your lunch and we'll see you at 2:30.

(End of session)

APOPS Session Two - IPv6 Operations

Wednesday 5 September 2007 1430-1600

PHILIP SMITH:

Can everybody take their seats, please?

HIDEO ISHII:

So, let's get the ball rolling and we'll start the session, the first topic of session two is IPv6 deployment. So for this presentation, we'll talk about deployment and we'll go from there. So let's start.

First Yves Poppes, and IPv6 deployment at Teleglobe. You can start.

IPv6 Deployment at Teleglobe - Yves Poppes

YVES POPPES:

Thank you very much. Good afternoon, ladies and gentlemen. A great pleasure to be in New Delhi. The last time I was in New Delhi was early 2005. There was an IPv6 session organised. Because IPv6 was one of the points on the agenda of Mr Moran and it was the sixth point, and little did I know back in early 2005, that one year later, VSNL would buy Teleglobe. So now I'm part of the family. IPv6 is something we've been involved in quite early from a Teleglobe perspective. And, of course, an internal question I get - "Did it pay off to have an early mover advantage?" I had a colleague who doesn't like IPv6 and I think he doesn't like me, who was asking the question last year, "How can you spend one third of your time on IPv6? And it only generates 1% of the traffic? So I suggest we should cut your end of year bonus." But I had some answers for him.

So what I would like to share with you are some ideas on the perceived drivers for IPv6 and there are quite a number of them and everyone has their preferred drivers. And the second aspect - are there really early mover advantages we can speak of?

First of all, to put things into context, we're a member of the Tata Group. We don't have too much time to explain the Tata Group so basically the Tata Group took an interest in VSNL, then decided to expand overseas. For that effect, they created VSNL International, which is based in Singapore. And with VSNL International, they made two - one is the title cable, which gave Trans-Pacific and Trans-Atlantic capacity in Europe. And then VSNL, which was acquired in February of last year. So basically we're now part of the VSNL International structure. So I report in to Singapore.

The result of this acquisition is that you have a variety of lines of business. Wholesale Voice was one of the major reasons VSNL bought Teleglobe. The entry zone is 21 billion minutes of international voice traffic. This is the biggest carrier in the world. Mobile - particularly being in America is the mobility side, all bodies essentially come in from GSM and CDMA. GSM is the European standard and CDMA is the North American standard. And global transport services, which is the overseas capacity is the contribution of this.

If we come back to our Internet - and we're all quite close to the subject - if we read the declarations of all in the industry. Internet proliferation, trillions of things connected and obviously it is a bit too small to connect all these things and give an address to everything. The Internet is becoming a key ingredient in our economic infrastructure, on the same level as electricity and roads. It's part of our social structure. We can wish Internet will go away but it will never go away. On the contrary, it's getting closer and closer.

And reading on the plane coming down here, 'Business Week', the last issue, they basically have their scenario, the issue is on the end of work as we know it. The teams of technology are outsourcing of globalisation, etc. Basically what they say is that we will have our billions of wireless sensors attached to buildings, retail products and etc. The same recurring theme.

As we all know, we come out of a major recession around the year 2000 with the famous dot com bubble and a lot of us were a victim of that phenomenon. Looking forward, a lot of thought was given where are the next billion revenue resources? Everyone agrees a number of factors have to be met. One is we need wireless broadband. Everyone is talking about IP convergence, convergence on the axis and the transport. We need multi-functional devices. Look at our cell phone, it becomes multi-functional. It has to be always on, always readable, nomadic. And good with security. All of us who are familiar with the IPv6 arguments will recognise a lot of this in there. And if we move around, we have mobile ad hoc networks. We have networks involved when we're on the plane. We have the notes moved. And the latest is MANEMO. One thing which has happened in our industries are the blurring distribution models. Basically, there used to be a time when you were in Telecom, or in music, or in printing, or in home entertainment and you had a discreet little niche. Everything is mixed up in an e-world. It is disruptive on our existing business models, be it from a carrier or an industry point of view. If we extrapolate in to the future, it's a kind of 'Star Trek' kind of scenario. Basically everything which can be materialised will be dematerialised. The transporter beam has not been an advantage yet but maybe we'll have enough addresses for each and every one of our atoms.

Now the one factor which is really spectacular is the growth of mobile. At the end of the first quarter of this year, we had 2.8 mobile devices and probably today, the three billion has been reached. Here in India, last month, I think 8 million cell phones have added to the network. 8 million is quite a number. Practically two times the population of Singapore. So if you look at that growth, it is incredible. I used to have the same slide two years ago and basically what I had there at the time was three billion would be reached in 2010. And I changed it to 2009, then to 2008 and now it's to 2007. Making a prediction is never easy especially when it's about the future.

Now mobile devices and Internet. In fact, when Vint Cerf was in Bangalore, he said the huge growth of the Internet and mobile phone users, not computers, the logic there is you have many more cell phones or personal devices that are computers. So, basically, you need the famous cell phone to be your computer, to be your end device. So you have a kind of convergence there on the end device which I think is quite clear, when you look at the iPhone. I think we even have a iPhone in the audience. It is a spectacular device. Now from our friends at Google, there's talk of a gPhone. Although they deny it exists. And Microsoft - I was looking up my shares and I have shares in RIM - if you have a Blackberry you know what it is. There was a rumour Microsoft would be RIM. So the shares went up a good 20%. I was wondering if I should sell them or hang on to them. So basically it's good business to see it all evolve.

If we continue on the mobile track, the mobile people every year at their main event, it uses to be at Cannes in France, but that became small, so they took it to Barcelona. The second edition in Barcelona. Basically, if you listen to what to what was said in Barcelona, they're waiting for the major breakthrough beyond voice and SMS. A Short Message Service has been one of the spectacular successes which nobody expected. It generates billions of dollars in revenue and it uses the signalling channel from telephony. It's amazing. And, of course, if you're talking about high-speed links, you have similar use where VoIP could destroy the revenues from some carriers, including some of our revenues, because we make money from roaming. So if I can forward roaming charges, why shouldn't I? So VoIP is something and VoIP over data, over mobile is something else. Of course broadband could trigger an explosion in use of social networking. We're all familiar with YouTube and things like this. Nokia thinks only a fraction of the email accounts use mobile. Only 5% of them. So there's market there. Of course the iPhone has been considered a threat by major operations, so they tried to react. I was reading yesterday that the rumour is that now Apple would announce a ring tone service, downloads on the iPhone and that could be an announcement next week. Again, everything is moving.

3G as mobile access. We mentioned that. It is growing by leaps and bounds. If we look at one of the components is IMS, famous Internet Multimedia Subsystem. There are conferences on the topic. People think lots of money can be made. I once labelled IMS as Internet Metering System because the mobile telephony people nearly threw eggs at me. They said we don't know how to bill. We in the mobile world know how to bill.

Now IP address-based billing. If you had a scenario where you billed someone on the IP address, it's better they have a permanent address. In the hotel upstairs, it's 10.10.something address. I'm behind in that. They would never be able to bill me a specific IP address service. The argument for IPv6 - if they don't care about IPv6, some argument will work because everyone wants to be able to bill their customer.

Data and mobility. I think the Japanese example is quite striking. If you look on this slide, since 2002, when 3G was introduced and you look at it today, the majority of these telephones are 3G. Just imagine what will happen in China soon when the 3G licences will be issued. There will be a real explosion again there in data.

Now what do they use their famous 3G handsets for? Every now and then you talk on that cell phone, email, Internet search, etc. Again, all data-oriented applications. So your mobile device will be your computer and your everything.

IP Telephony. It is disruptive. It has been disruptive for our telephony revenue at Teleglobe and it continues to be. If you cannot beat them, join them. We bought one of the major VoIP international providers. If you look at it today, in North America, we have about 400 providers, and BB in Japan is the biggest one. The numbers still increase and with the growth of Google and Microsoft and eBay, again, they will change and continue to change the world. And Skype, we all know this phenomenon. Kazar is behind them as well. Six months before eBay bought Skype, someone said they would not pay $100,000 for Skype and six months later it was sold for $2.6 billion. Again, to make predictions is always dangerous.

Windows, as IPv6 deployment catalyst, Windows Vista, it gives preference to IPv6. If you want application like call-up operation, you need to be able to reach the address on the other sides. And I like a sentence that was said that the American Forces Conference, "We are betting our business strategy on IPv6 and this." IPTV is another catalyst. Those of you who go to IPTV conferences will have seen the sessions. ComCast ran out of their private address space. And basically they're looking at IPv6 and to deployment. A little statistic I gave here, I got from the 'Economist'. And basically the interesting part is you have the IPTV component which is local distribution of television and then you have the Web TV component. International carrier, I like the Web TV component. I had a meeting here and some of the Indian TV producers want to have the Indian communities overseas watch their programs using Web TV. In Canada, for example, we have quite important Indian communities. So this will generate a lot of potential traffic.

Number of networkable devices. If one looks at this statistic, which was in a presentation in early 2005, then if you accept at 17 billion networkable devices, I think the four billion IPv4 addresses are not enough. If you include others, you reach much more. These figures could be optimistic but if you look at the hand-held subset categories, or the cell phones, they projected it at 2.6 billion. It's 3 billion today. So, basically, that statistic at the end of 2004 was at least in that category too conservative.

RFID - another one of this phenomena, I think some of you must follow the efforts in Korea. So basically the idea is to associate an IPv6 address of sorts. Everyone had thought it would go very fast but take-off has been slower than expected. But again it's getting there. And, again, why - traceability? The major ones - bank notes, passports. The Swiss came up with that idea to have an RFID in every bank note so you will go through Customs one day and they will say, "Why do you have $264 in your pocket?" Prevalence of digital access is another one of these drivers. DSL cable, WiFi and Wi-Max phenomenon.

The national policies is quite an important factor. We were mentioning the 10-point agenda here in India. Of course in China we have this NGI project. Korea and Malaysia. Malaysia are pushing for the IPv6 adoption. In the US, the Department of Commerce is pushing for 2008. Why is everyone pushing? It's basically to improve the contribution of information, technology to the gross domestic product.

National policies on the defence side, I think there again military wants memos because they want to have automatic configuration.

What does IPv6 bring to the table? It solves address shortage. It restores communication. It makes roaming easier. Battery life is even better. And with the configuration, you have mobile networks, etc. And permanent addresses, we have all the advantages we took for granted in telephony.

Early mover advantage - from a Teleglobe point of view, we moved early. One of the reasons was we were involved in the NGI in 1995, when the old Internet, became commercial. I had a call and he said, "You have a new cable." And in the Atlantic, "Why don't we use that for the next generation of research network?" I was part of a board in Canada. It's the first time I heard about IPv6. They said we will join together in Chicago with another company. The rest is history.

So basically, if you look at it today, we have that famous global footprint which we got through the Tyco way. We're running the global network, which is dual stack, before and v4 and v6. Surfing the globe is one of the major arguments because you would remember the Taiwan earthquake in December last year, when basically most of the cables were cut to the west. All traffic had to go to the east. Two years before, there was the earthquake in Algeria when all the traffic between India and Europe, for example, had to go via the US and then to Europe.

Now there's an explosive growth and that's thank you to YouTube and MySpace. 4-5 years ago nobody would have anticipated the impact of user-generated content on that. So basically that's results in about 600+, 700+.

Now the bonus is IPv6 in the core. Because only the traffic growth could justify this expansive system. So thank you for YouTube and user-generated contacts.

Now if I look at it as a tier 1 carrier, IPv6 is still minimal. Traffic growth depends on adoption in tier 2 networks. The success of the tier 2 networks depends on the application. They're looking to Windows Vista and other operations. And would it be Vista or the exhaustion of IPv4 addresses or a combination? We don't know.

I had that colleague of mine who didn't like IPv6 and the time I was spending on it. So the answer I gave to him at the collective review was, "It gave high visibility and an early advantage. It gave a different move in the marketplace. If I only sell IPv4 and you sell IPv4 and IPv6, where will they get their service from - the one who was IPv4 and IPv6, even if they don't use IPv6 today." If one looks at the RFQs we had last year, we had about 60 major requests for transit moves. About half of them gave points to IPv6 and about 10 of them made it mandatory. So it was an advantage to support IPv6. So my colleague, he didn't argue anymore. But the next step is to stimulate growth of IPv6 traffic. Do you still spend one third of your time on IPv6 and still only generate 1% of the traffic." I have to be careful there.

In India, it has started on the AS4755. A set here is a - it is available as we speak. So thanks to Gaurab, we have set up a network and more on India at the next IPv6 summit, which it is confirmed for December 12 and 13. Please come and visit us there. A lot of new things about IPv6 in the region should be said and announced there.

Now today we have Gaurab and the IPv6 with SANOG. In conclusion to some thoughts, I think our human activity is effective by telecoms. No doubt about it. I think we have digital lifestyles, we make friends, even get married on the internet. Business processes from design to consumer or are Internet dependent. Some people are addicted to the Internet. In Amsterdam, they have a detox centre for addicts. A man was spending 18 hours a day on the Internet. He didn't eat anymore. I think there will be all networks in motion. I think we will be moving around with lots of information and communicate and pick up its speeds. Of course, we will be ad hoc networks, so where does the carrier make a revenue if we are all part of the big network? The downside with Ambient, Pervasive, whatever you call them networks, there will be no place to hide anymore. So if we have an advantage, don't forget whatever advantage you have, someone will take it away from you. Thank you.

APPLAUSE

HIDEO ISHII:

Any question? Thank you very much.

Next we have Bijal Sanghani.

IPv6 Deployment at FLAG Telecom - Bijal Sanghani

BIJAL SANGHANI:

I'm going to give you an overview of the FLAG network. We've got multiple STM backbone links which mostly sit on the cable system. It's diversely routed and fully redundant between core PoPs. We're in the process of upgrading to multiple 10-gig and these will sit on the T640 platform. IP transit is the main drive for capacity growth on the network. Our customers are typically ISPs, specifically around the Middle East and Asia. FLAG has big expansion plans and you can follow the links on the slide for more information.

This is our fibre optic networks. All these sites are now live. You probably can't see very well from the back. I'm sure the slides will be up on the website.

This is our global IP network and our peering points. Apologies, because this slide is slightly out of date. India Mumbai is missing on there for a start, and, yeah, I think that's the only one that's missing from there.

OK, so a bit about our building blocks. We've got a single-area level 2 of IGP. Metric is based on distance and round trip time. We use this as our primary method to control traffic flow. We do sometimes have to adjust the metrics artificially; for example, if there's cable cut or something like that. We've got a global AS number, 15412. Communities are used extensively and all routes are tagged when entering the network. You can find a list of the communities that we use on the RIPE Whois database. Our prefix filtering is done by AS-set, so customers need to register their routes with the appropriate registries and maintain their AS-sets. Routes are not accepted if they're not registered.

Both LDP and RSVP is enabled on the network. All traffic is labelled switched and follows the IGP metric. RSVP is used for load balancing for special engineering if needed, again during cable cuts for example, to shift traffic and reroute. We have a primary LSP set-up with fast reroutes so, if a link goes down, the next-best route is used. To control LSPs and multiple links, we use link colouring, bandwidth, least fill, band rid and re-optimise.

The services that FLAG offer include IP transit, direct routes, - that's our customers and peer routes only - global Ethernet, VNAP, layer 2 VPNs, based on Kompella, layer 3 VPNs, multicast and IPv6, which is what I'm going to talk about next.

So IPv6. We talked about it for ages but had all the excuses not to deploy it. There was no demand, no business case, no time, too busy, too hard to do. We finally completed the implementation in December last year and this was due to a customer demand in Asia. Since then, we've had a few more customers connected in Asia and we've seen growing interest in the Middle East.

We currently offer IPv6 transit to our existing customers for free. It's offered globally and customers can connect locally. The support is relying on a few senior engineers offering best effort with no SLAs and support is done via e-mails only.

This is our allocation, which we have from RIPE. And this has been broken down in various /48s for customers and internal assignments. The /48 it's have then further been broken down into /64 s for our back-to-back backbone links and customer WAN links. All Flag PE routers are dual stacked, so customers can connect without additional physical or logical connections or a tunnel. However, we do support tunnels if needed. Customers can choose to run v6 and v4 in one connection or run v6 on separate logical connections.

We use the same routing policy and BGP communities, except dampening and black hole is not supported yet. We basically tried to keep everything the same as v4 as much as possible on the network.

We've got a congruent ISIS topology with v4 and have a separate iBGP mesh for v4 and v6. This makes it easier to apply different policies to v6 if need be. The current implementation is native IPv6, so v6 traffic is not labelled switched.

With Flag's NGN expansion plans and having new P player routers, we're probably going to go for a 6 PE design in the future. Currently we don't have any tools for prefix filtering or config generation. This is all work in progress. So customer prefixes are updated manually when we get e-mails sent to that e-mail address that was on the other slide.

Monitoring is done via MRTG or using CLI on the routers. We also have Are bore supported in v6 as well. The current version of JUNOS that we're running does not support v6 cflowd export so we don't have any stats for v6 at the moment. There's no v6 content on there and I'm not really sure where it is.

OK, we rolled out v6 on all our routers with no problems. The actual work took about three days. The most consuming part was putting the IPv6 addresses on all the backbone interfaces and establishing the iBGP sessions. New configuration was needed on the routers. This included interface addresses, firewall filters, policies and BGP. It actually took longer to write the design documentation, and planned work procedures and worrying about it than actually doing it. Juniper supported v6 for quite a while. However, we were running an older version of cflowd, which had a security bug so we had to upgrade that before deploying. This delayed the deployment. The original plan was to have v6 on the network by August last year. However, saying that, I'm not sure what the consequence would have been if we had had v6 on the network and then found out there was a security bug.

Sprint's existing route connected to us was not v6 enabled, so we needed a tunnel to the nearest v6-enabled router. So for this, on our network, we need ad Juniper tunnel PIC. There were problems with the initial BGP session, as we saw, that it kept flapping. We then realised that was due to the M TU size problem. So even though we had path-mtu-discovery configured, sprint had to lower the MTU on the tunnel interface. We found that it stabilised. Another problem was that when a new IP backbone link is added, the IPv6 address had to be configured at the same time as the IPv4 address, otherwise the IPv6 packet forwarding will break and all iBGP sessions using that backbone or traversing it will drop.

We also needed to allow ICMPv6 on the management filters for loO for Ethernet connections, for example, on connections to the Internet exchanges, this had to be done. This was a change in policy from how we do it for IPv4.

We needed to adjust the firewall and bogon filters a few times after we deployed IPv6 as we didn't realise a site local address had been' placed by a unique local v6 address. I think this came out in the IETF meeting just before we deployed v6 on the network. That's something we need to keep on top of for the future.

As for the future of v6 in FLAG, we'll carry on supporting basic IP transit. We have an open peering policy at all the Internet exchanges that we're at. We're particularly interested in peering with academics and researchers in hope to see more v6 traffic on the network.

Unfortunately, it's unlikely that we'll do much more with v6 at the moment, due to other projects and since it's non-revenue-generating, it's not the company's primary focus.

We would like to offer SLAs on the service and we'll be able to do this once our NOC is fully trained and we have their support.

We'll keep tracking v6 developments and keep filters updated and we're ready to serve customers when they need it.

Currently, we don't have any plans to offer any transition gateway services.

IPv6 is still a specialist subject in FLAG. Been hard to train everyone to support v6 in the NOC, even though we've tried to keep everything as similar to v4 in as possible, it's still very new to the operations. We're not really sure how the customers are using it, whether it's just a regulatory requirement or if they're actually using it for some real applications. We would like to get v6 connectivity to the corporate IT network, but that would probably be a bigger challenge than rolling it out on to the production network. I think we've been brave, but we do worry about our filters not being updated and possible v6 bugs that might be around. Unfortunately, we don't have enough time to keep track of all the changes on top of everything else that's going on, as we don't have a v6 person dedicated v6 person inhouse.

Finally I'd like to say thanks to George, Wai-Kay, Lillian and Che-Hoo for making IPv6 a reality in FLAG.

Any questions.

MARK SEWARD:

I guess this applies to both equally. Have you done any analysis into transition traffic on the networks? That is, you were saying before, there's a small amount of v6 traffic that is carried natively across the network. Has there been any sort of further looking into transition of them, so any 6to4 traffic, any awareness of how much of your v4 traffic is actually carrying and encapsulating v6 traffic. Is that being potentially used as a predictor of future growth?

BIJAL SANGHANI:

We're seeing such a low amount of v6 traffic.

MARK SEWARD:

In terms of what's being natively carried across the network?

BIJAL SANGHANI:

In terms of v6 traffic that's actually carrying v6 encapsulated. That's not been looked at?

BIJAL SANGHANI:

Yeah.

MARK SEWARD:

I think that's would be quite interesting. I thought I'd ask.

Thank you.

HIDEO ISHII:

Any other questions?

OK. Thank you very much.

OK, next presentation is IPv6 migration challenges for the large networks, from Aruna Pidikiti, from Bharti.

OK.

IPv6 Migration challenges for large networks - Aruna Pidikiti

ARUNA PIDIKITI:

Good afternoon to everybody. I am Aruna from Airtel and I am the manager of network operations, based out of Delhi.

Today my topic is IPv6 migration challenges for large service providers. This I made in context to the current data network.

I'm going to talk on brief overview of our Airtel, what are the drivers for us, driving us towards the IPv6, what are the migration challenges, the design considerations which we are going to take in and the deployment plans.

We are the largest telecom service provider, that is the first service provider which is having footprint in 23 circles, almost all circles in India. We have 42 million subscribers and we have fixed lines in 94 cities across the country. We have used VSAT network. This is basically to reach towards remote locations. And we have fibre network across the country approximately of 40 kilometres. And we have international connectivity through i2i, SEAMEWE-4, for both services. We also provide integrated solutions for using this entire infrastructure.

This is the product portfolio of Airtel where we provide the data services, mobile, collocations, we provide the integrated solutions. So this I will be explaining later specific to the data services.

The infrastructure which we have, approximately we have 120 PoPs across the country having the additional capacity of 25 gig and peering with all major Tier-1 service providers and we also have peering with NIXI for our national capacity and also major local ISPs.

This is about our ISP infrastructure. We have the MPLS infrastructure approximately of 120 locations and having five international PoPs, we do provide the L 2 and L3 VPNs, we have NNIs with various carriers, like Singtel and Hong Kong. We do provide multicasting and QoS.

We have the ATM and the FR infrastructure. We provide the wholesale bandwidth to the carriers. We also have the downstream carriers who further sell bandwidth to their subscribers. We have the VSAT network through which we provide the connectivity to remote locations where our network is reachable, either to fibre or copper. Through VSAT, we also provide the L3 VPNs and this MPLS and VSAT, both networks are integrated. We have the metro Ethernet in all major cities, WiMAX for the access, GPRS and EDGE.

So this is about our services, which we currently have in our network, so based on these services, what are the drivers for Airtel for the IPv6 integration?

This is basically the predicted broadband growth, what is the kind of network we have. We have all kinds of access. We have dial-up, dial-up services, we have DSL where major growth is there, the fibre or the connectivity through copper, we have the Ethernet customers, WiFi and also WiMAX.

Where the growth for our network currently is, we have 1 million subscribers and the growth month on month is approximately 35 to 40 gig.

Moreover, on this broadband, we need to do the more IP s to the subscribers. Why do we need to do the more IPs? Because of the services. Like every home user know wants to have an IP-Phone, to have IP for the TV and for the music instruments and more appliances, so that is why the need for more IPs.

In summary, why we want, why the integration IPv6 integration is required? Because IPv6 address pool exhaustion, it is going to be deployed, increasing broadband users, impact - the Indian Government is more focused on the broadband subscribers so this requirement is increasing day by day.

The IPTV services, which is a potential market in the Indian market, class 5 VoIP and other applications, like mobile IP, which is going to come in future. So basically these are the drivers, which is driving us towards the integration of the IPv6.

The next one is what are the migration challenges in if we want to migrate our network towards the IPv6, what are the migration challenges we face?

First one, we need to assess the network. We have to assess in every aspect, every element of the network. We need to divide it into mainly two phases, one is infrastructure, the devices there in the network. The second phase is the servers, OSS, all these kinds of applications have to be assessed. And by assessing, we have to see what is the current hardware we have, what is the memory sizes, what are the kind of interfaces we have and what is the current CPU utilisation, because when you integrate, the memory and the CPU utilisation may increase, because when you are integrating from IPv4 to IPv6, your look-up will happen in 32 bit. You need to see all these aspects also. What is the current software version, what is the version you have to upgrade? Because this is not the equipment which we have deploying now.

The network we have is almost five or six years. That means from the day we have deployed the network element, those are the elements we need to assess. What is the capability we have? Do we have the capability? This is the assessment we need to do in each part of the network. In the infrastructure, again we can categorise, both core and access.

In the infrastructure, we have the core network - every large service provider has core and access. And again we see in the core. When we integrate, do we require this core has to be IP aware or IP unaware? If it has to be IP aware, what is our infrastructure we have? What kind of integration we can do? Whether the existing elements can support it. Did we need to do the upgrade? In case you have to upgrade, what is the investment plan? Everything has to be assessed and it is a big challenge, actually. If we take from the - if we want to make the core unaware, to start with the case one, probably we need not do any changes, we can start only with the access. At the access also, access actually starts from the customers. If the customer is ready right now to have an IPv6, because the applications, everything starts from the customer, so if the customer is IPv6 aware, also the customer has to be ready.

If we go for the native IPv4 or IPv6 only on the edge, then both see that the customer is equipment and the PE, which is the edge equipment, both have to be IPv6 capable. That means we need to see the compatibility of both the hardware and the software and the routing protocols in this aspect.

So specific to our network, we have an access network through broadband, where, if you see the elements, the customers, equipments, connected on the DSL modems, further connected to the DSLAMs, further connected through the metro Ethernet on the broadband RSA. So whenever we are integrating, we want to integrate the challenges we face, we need to see each and every element. It's not only the customer equipment. We have more routers, we have - what are the switches in between. Whether the DSLAM is supported or not. Whether the broadband RAS is supported or not. The broadband RAS further has to talk on the backend for the investigation purpose and the billing purpose. We have the billing based on the utilisation, based on the time, all these mediations, billing service based on the backend or not. And dial-ups. Dial-up also on the RAS that we have. Currently it supports the IPv6, so in this way every element has to be assessed and see what is the compatibility we have when we want to indicate or migrate to IPv6.

Similarly, the approach which I have taken is basically on the service line. When we want to integrate, it's better to approach the service lines, the entire network in one room. So the first thing is taking the access part - broadband, VSLANs or the dial-up. The next one is the IP TV. If we want the IP TV network to be IPv6 network, then is it supported on the set-top boxes and now we have various vendors on the set-top boxes, whether these boxes are supported, is it supported with OSS and BSS and are the DSLAMs supported because we have to enable multicast on the DSLAMs also. These are the elements which we need to see.

Then the Voice over IP. I believe the largest service provider has always these kinds of services. So Voice over IP, whether the soft switches are capable. If not, what is the applications we have to do, what is the software required, the IP phones and the media gateways and the DHCPs, the NMS services, the DNS and the BSS, OSS part, everything has to be assessed before we start any integration on the service.

One aspect is the network, which is the network infrastructure. We assess it on the services. We have seen what are the challenges we face or what are the aspects we need to look into when we want to integrate infrastructure. So for a large service provider, infrastructure is not important. The important aspect is the monitoring tools. What is the kind of NMS we have. What is the kind of tools we have for customers, logging, what kind of reports we provide to the customer, what are the troubleshooting methods we have. For all these, the NMS is a very important aspect. If we see the current NMS, which we have, then we have DHCP, so it creates the topology based on IPv4. So this NMS is capable of having the IPv6. Can it show the topology based on IPv6? So that also we need to see.

And the troubleshooting system, because there is a capacity to have an automatic troubleshooting system. What are the tools we have in terms of the network protocols, whether the SNMP supports that. These are the main things. These NMS applications, whether it supports - the dual stack is supported on these NMS applications. That is the one aspect which need to see and this is the major challenge in terms of the network monitoring and the management.

The other important aspect is network security. So we will be having the firewalls in our network or the appliances in our network, whether these firewalls are supported right now, what is the upgrading we have to do. So it is basically a broad thing where every element has to be mapped and see the compatibility, what is the operations we have to do in that aspect. And threat protection. The packet filtering, whatever is happening right now on the fine four is the same kind of packet filtering happens in the same network or we need to upgrade any infrastructure in our network, some elements have to be upgraded and the secure connectivity, can we connect on the IPsec or the hardware where we have the IPsec connected and the hardware based on IPsec, can we use the same? Probably not. In a few hardwares, we need to change GPSAT. That is one of the major challenges in terms of the security.

So as I was explaining, OSS/BSS a big challenge, based on these servers or the mediation services, the billing servers which we currently have on IPv4. We need to see the licence part and see how we make the dual stack enabled. And one important thing is legal interception. In India, actually, every ISP service provider should have legal interception facility. That means any packet which is going out of India should be monitored. So the current existing infrastructure may not be for Airtel network, it is for all the networks, is capable of having a dual stack, whether it is compatible of having the IPv6. And the customer portal, which we do to the customer, probably we may be giving the services still now. After the integration, we should be able to do the same services to the customer without any compromising on the quality and the service.

And as I explained on the FCAP, basically what is our configuration, what are the processes, accounting and the performances.

What are the other challenges? As I explained, though we may be ready for overcoming all these challenges, but the customer is ready? Why do we need to do all this investment if the customer is not ready? Probably one or two customers will be ready for the IPv6. Do we need to put this kind of investment only for two or three customers? What are our operational issues if we migrate from IPv4 to IPv6, even with the existing infrastructure also, within a stabilised IPv4, we will be having many operational issues in terms of the upgrade issues, in terms of software issues, all the down times or the card issues. Similarly, the same issues can happen in IPv6, probably even more because we don't know how that works. And the training. People have to be trained on this. Without training, we will not be able to maintain the services.

So these are the major challenges. If one has to migrate from the existing IPv4 to IPv6. So in every aspect, maybe a few services I have not covered because, as a large service provider, these are the major services. We also have the GPRS network. If we have to integrate the GPRS network, we need to see what is our infrastructure, how do we upgrade that and give those elements. Like any service provider, we see what are the challenges he faced when he has to upgrade.

Based on these challenges, we have considered few design objectives for our network.

Basically this would be the overall scope of the IPv6 deployment. The scope starts from the services, as I explained and also the access on the core, what kind of strategy we have to deploy and on the services, IPv6 services, once we deploy this IPv6 integration, what kind of services we can provide and what are our routing protocols, what kind of protocols we can use.

So if we see the overall scope, we will be approaching as a service device.

Our plan is basically to do in three phases. The first phase is do it at the access level. Then slowly come to the core level, make the IPv6 within the core infrastructure. Then the next one is inter