Debugging eduroam at TNC2010

Yesterday Matti was having trouble with Helsinki University user accounts and the eduroam network here at TNC2010. Together with few eduroam guys (Stefan Winter and Miroslav Milinovic) and Heikki Vatiainen from our company we tried to find out why the authentication process did not seem to go through. What made it harder was that for example Radiator does not log enough information about failing authentication process unless it clearly ends in success or failure. Increasing log level to debug levels of course helps and we were able to see that the Helsinki University RADIUS server did send response messages to authentication requests even if the process did not go through.

Other servers in Finland worked fine (I have been able to use eduroam the whole time with my TUT accounts) and only difference we first found was that Helsinki University certificate was bigger than the ones in some of the working organisations. This lead us to suspect RADIUS message fragmentation problems somewhere in between European Top Level RADIUS and Matti's device. We then remembered the one Radiator configuration directive, which we found was missing from Helsinki University RADIUS server configuration.

Adding EAPTLS_MaxFragmentSize 1024 solved this problem so if you are having similar problems with eduroam and your home organisation is running Radiator RADIUS, we suggest checking at least this setting found in Radiator manual:

EAPTLS_MaxFragmentSize

For TLS based EAP types such as TLS, TTLS and PEAP, this optional parameter specifies the maximum size in octets permitted for each TLS message fragment. Defaults to 2048, but many EAP clients, routers and wireless Access Points have limitations that require EAPTLS_MaxFragmentSize to be set as low as 1000 or less. Setting this number too small can result in excessive RADIUS request round trips during EAP TLS authentication, slowing down the authentication process. Setting this number too large can result in failure to complete TLS authentication for some types of clients and devices.

Finding rogue IPv6 routers on Mac OS X

In one of the larger wireless campus networks there was a problem of an annoying host advertising 6to4 (2002) and fec0 prefixes to a network segment which already had an official IPv6 router. This is the same kind a situation as rogue DHCP server in IPv4 network. All traffic is sent to go through the advertising host and if that can either route or drop the traffic making IPv6 services slow or unusable. There exists few extensions for IPv6 to secure router advertisements and only accept proper ones, but those extensions are rarely implemented in the mobile devices.

So to find the owner of the misbehaving host, one option is to find the IPv4 address from the 6to4 prefix and inform NOC (Network Operations Center) about it. In 6to4 addresses the original IPv4 address is part of the IPv6 address so we can find out the corresponding IPv4 address this way (example IPv6 prefix 2002:c0a8:2a2a::/48):

% printf "%d\n" 0xc0
192
% printf "%d\n" 0xa8
168
% printf "%d\n" 0x2a
42

The IPv4 address corresponding to 2002:c0a8:2a3a::/48 prefix is thus 192.168.42.42. The IPv4 address is often enough to find the host and its owner, but in large wireless networks the IPv4 addresses may get reassigned so also the time of the problem must be recorded. Then the host and owner can be checked from the DHCP server logs or from the wireless network management system such as Airwave.

There exists also a way to identify the host faster and that is to find out its ethernet mac address. In IPv4 there is ARP, which is used to find out the mac addresses of the corresponding IPv4 addresses. In IPv6 the similar protocol is called Neighbor Discovery Protocol (NDP). The problem was where and how to find this information. In Linux it is possible to use ip utility for this (the addresses in these example are not related to the 6to4 culprit):

% ip -6 neighbor list
fe80::224:36ff:fe9d:c1dc dev br0 lladdr 00:24:36:9d:c1:dc router REACHABLE

It took me for a while to find out what I could use on Mac OS X, but the manual pages hinted that Mac OS X's IPv6 stack conformed to the NetBSD implementation documentation found at:

• http://www.netbsd.org/docs/network/ipv6/
  

The command needed for neighbor discovery protocol control on Mac OS X is called ndp. With this command it is possible to display and manipulate neighbor discovery protocol tables and find out the corresponding ethernet mac addressed for default router IPv6 addresses (listed with netstat -nr):

% netstat -nr
Internet6:
Destination                             Gateway                         Flags      Netif Expire
default                                 fe80::212:3400:9c56:7890%en1    UGc         en1
% ndp -a
Neighbor                        Linklayer Address  Netif Expire    St Flgs Prbs
fe80::212:3400:9c56:7890%en1    0:12:34:56:78:90     en1 23h59m9s  S  R

The more hardcore IPv6 specialists may read the ethernet mac address directly from the link level IPv6 address. The problem is that the link level address may not be always formed from the actual linklayer address so this method is preferable and also a bit more friendlier to user