Announcing the ICSI Certificate Notary

We are happy to announce the ICSI Certificate Notary today. This service provides near real-time reputation information on a large number of TLS/SSL certificates seen in the wild, collected continuously from a set of partner network sites. The notary’s data includes the time when a certificate was first and last seen, and whether we can establish a valid chain to a root certificate from the Mozilla root store.

Since the beginning of this year we collaborate with operations at about ten large network sites to passively extract certificates from their upstream traffic using Bro. This has allowed us to build a certificate database that now comprises roughly half a million unique web certificates from over 8 billion connections, representing the activity of estimated 220,000 users. (In fact, we have collected 7 million unique certificates but the majority is non-web activity and hence excluded from the notary.)

You can use the service by sending a DNS request for an A or TXT record to:

<sha1>.notary.icsi.berkeley.edu

The token <sha1> represents the SHA1 digest of the certificate to query, which you may find when consulting your browser for details about a certificate. For A record queries, the result comes back either as the address 127.0.0.1 to indicate that our data providers have seen the certificate, as 127.0.0.2 if we could recently validate the certificate against the Mozilla root store, or NXDOMAIN if we have not seen the certificate. For TXT record queries, the notary returns key-value pairs with more details. Here is an example reply:

"version=1 first_seen=15387 last_seen=15646 times_seen=260 validated=1"

For further details, usage instructions, and background reading, please visit the notary website at http://notary.icsi.berkeley.edu. We much appreciate your feedback at this early stage, both positive works-for-me notices as well as problems and suggestions for improvements.

IMC'10 Paper on Illuminating Edge Networks

Earlier this month we presented the ICSI Netalyzr at the Internet Measurement Conference in Melbourne, Australia. The Netalyzr is a public edge network measurement and debugging service that evaluates the functionality provided by people's Internet connectivity. Its tests include outbound port filtering, hidden in-network HTTP caches, DNS manipulations, NAT behavior, path MTU issues, access-modem buffer capacity, and growing IPv6 support and performance. The paper is available here:

• Christian Kreibich, Nicholas Weaver, Boris Nechaev, and Vern Paxson. Netalyzr: Illuminating The Edge Network. Internet Measurement Conference, 2010, Melbourne, Australia. (bib)

The Netalyzr has been one of our major research efforts over the past two years, and we're thrilled by the popularity it has gained since we launched it—to date, Netalyzr has collected 160,000 sessions from 6,800 different organisations in 190 countries: The study is ongoing, so visit the Netalyzr website and run it yourself!

ICSI Netalyzr leaves beta

Today we are taking the ICSI Netalyzr out of the beta stage. Among the changes we are rolling out are:

• New tests. We now provide a path MTU test, IP fragmentation support, improved DNS examination, and look up additional names. Besides the client-side transcript you can now inspect the server-side one, which is useful for debugging highly troubled sessions. In addition, we have improved the overall robustness of the existing tests.
• Interface improvements. A frequent complaint we received was that the results summary is overwhelming. As a first step to improve the situation, you can now selectively show or hide result summary detail. On the summary page, you find clickable plus/minus symbols that will expand/collapse test results on the entire page, in a particular test class, or on a particular test. When you first arrive at the summary page, any issues we have noticed remain expanded by default.
• Updated info pages. Each of our tests comes with an info page, available by clicking on the test's name (such as "Path MTU" in the above). We have given those info pages a makeover, which will hopefully make them easier to understand and more useful to less technical users.

We hope you will enjoy the new Netalyzr. Many thanks to everyone who has tried out the tool in the past!

Introducing the ICSI Netalyzr

Today we're very happy to announce public availability of the ICSI Netalyzr. Our goal was to build a service that shows you in detail what's up with your network connection, whatever network you might find yourself in, whenever something's not working, or when you're simply curious. The numerous tests conducted by the Netalyzr include HTTP proxy discovery, HTTP caching behavior, NAT detection, TCP & UDP port filtering, DNS resolver behavior, IPv6 connectivity, connection latency, bandwidth, and buffer properties, and more.

All you need is a Java-enabled browser and a visit to http://netalyzr.icsi.berkeley.edu.

We hope you'll find the site as useful as we do. We're very keen to hear your feedback, whether it's interesting results, suggestions for improvements, or any issues you've encountered.

Go forth and netalyze!

How to Report a Bro Problem

Generally, when you see Bro doing something you believe it shouldn't, the best thing to do is opening a ticket in the Bro tracker, including information how to reproduce the issue. In particular, your ticket should come with the following:

• The Bro version you're using (if working directly from the Subversion repository, the branch and revision number.)

• A small trace in libpcap format demonstrating the effect (assuming the problem doesn't happen right at startup already).

• The command-line you're using to run Bro with the trace. (Please run the Bro binary directly rather than using the bro.rc wrapper from the BroLite environment.)

• Any non-standard scripts you're using (but please only those necessary; ideally just a small code snippet).

• The output you're seeing along with a description what you'd expect Bro to do instead.

• If you encounter a crash, information from the core dump, such as a stack backtrace, can be very helpful. See below for more on this.

It is crucial for us to have away of reliably reproducing the effect you're seeing. Unfortunately, reproducing problems can be rather tricky with Bro because more often than not, they occur only either in very rare situations or after Bro has been running for some time. In particular, getting a small trace showing a particular effect can be a real problem. In the following, I'll summarize some strategies to this end.

How Do I Get a Trace File?

Since Bro is usually running live, coming up with a small trace file can turn out to be a challenge. Often it works to best to start with a large trace triggering the problem, and then successively thin it out as much a possible.

To get to the initial, large trace, here are few things you can try:

• Capture a trace with tcpdump, either on the same interface Bro is running on, or on another host where you can generate traffic of the kind likely triggering the problem (e.g., if you're seeing problems with the HTTP analyzer, record some of your Web browsing on your desktop.) When using tcpdump, don't forget to record complete packets (tcpdump -s 0 ...).

  You can reduce the amount of traffic captured by using the same BPF filter as Bro is using. If you add print-filter to Bro's command-line, it will print its BPF filter to stdout, which you can copy over to tcpdump.

• Bro's command-line option -w <trace> records all packets processed by Bro to the given the trace file. You can then later run Bro offline on this trace and it will process the packets in the same way as it did live. This is particularly helpful with problems which only occur after Bro has been running for some time. For example, sometimes crashes are triggered by a particular kind of traffic only occurring rarely. Running Bro live with -w and then, after the crash, offline on the recorded trace might, with a little bit of luck, reproduce the the problem reliably.

  However, be careful with -w: it can result in huge trace files, quickly filling up your disk. (One way to mitigate the space issues is to periodically delete the trace file by configuring rotate-logs.bro accordingly.)

• Finally, you can try running Bro on some publically available trace files, such as anonymized FTP traffic, headers-only enterprise traffic, or Defcon traffic. Some of these particularly stress certain components of Bro (e.g., the Defcon traces contain tons of scans).

Once you have a trace which demonstrates the effect, you will often notice that it's pretty big, in particular if recorded from the link you're monitoring. Therefore, the next step is to shrink its size as much as possible. Here are a few things you can try to this end:

• Very often, a single connection is able to demonstrate the problem. If you can identify which one it is (e.g., from one of Bro's *.log files) you can extract the connection's packets from the trace with tcpdump by filtering for its 4-tuple of addresses and ports:

  tcpdump -r large.trace -w small.trace \
     host <ip1> and port <port1> \
     and host <ip2> and port <port2>.
  

• If you can't reduce the problem to a connection, try to identify either a host pair or a single host triggering it, and filter down the trace accordingly.

• You can try to extract a smaller time slice from the trace using the TCPslice utility. For example, to extract the first 100 seconds from the trace:

  tcpslice +100 <in >out
  

  Alternatively, tcpdump extracts the first n packets with its option -c <n>.

Getting More Information After a Crash.

If Bro crashes, a core dump can be very helpful to nail down the problem. Examining a core is not for the faint of heart but can reveal extremely useful information ...

First, you should configure Bro with the option --enable-debug and recompile; this will disable all compiler optimizations and thus make the core dump more useful (don't expect great performance of this version though; compiling Bro without optimization has a noticeable impact on its CPU usage.). Then enable core dumps if you don't have already (e.g., ulimit -c unlimited if you're using a bash).

Once Bro has crashed, start gdb with the Bro binary and the file containing the dump. (Alternatively, you can also run Bro directly inside gdb instead of working from a core file.) The first helpful information to include with your tracker ticket is a stack backtrace, which you get with gdb's bt command:

gdb bro core
[...]
> bt
....

If the crash occurs inside Bro's script interpreter, the next thing to do is identifying the line of script code processed just before the abnormal termination. Look for methods in the stack backtrace which belong to any of the script interpreter's classes; roughly speaking, these are all classes with names ending in Expr, Stmt, or Val. Then climb up the stack with up until you reach the first of these methods. The object to which this is pointing, will have a Location object, which in turn contains the file name and line number of the corresponding piece of script code. Continuing the example from above, here's how to get that information:

>up
>...
>up
>print this->location->filename
>print this->location->first_line

If the crash occurs while processing input packets but you cannot directly tell which connection is responsible (and thus not extract its packets from the trace as suggested above), try getting the 4-tuple of the connection currently being processed from the core dump. To this end again examine the stack backtrace, this time looking for methods belonging to the Connection class. The connection class has members orig_addr/resp_addr and orig_port/resp_port storing (pointers to) the IP addresses and ports respectively:

>up
>...
>up
>printf "%08x:%04x %08x:%04x\n", \
    *this->orig_addr, this->orig_port, \
    *this->resp_addr, this->resp_port

Note that these values are stored in network byte order so you will need flip the bytes around if you are on a low-endian machine (which is why the above example prints them in hex). For example, if an IP address prints as 0100007f, that's 127.0.0.1.

Making Sure Your Bro Code Does Not Leak

This is for people hacking on Bro's C++ source code. Internally, Bro's memory management is pretty complex and when writing new code it's unfortunately pretty easy to introduce memory leaks. If Bro's memory footprint keeps growing over time (hours or days), and there's no script-level state table to blame for what you see, it might be a leak in Bro's C++ event-engine.

Bro has been in use for many years and we are pretty confident that there aren't any major leaks left in the more common code paths these days. However, there's also code which isn't used very regularly and and it still happens that people run into leaks in these parts.

But most importantly, for new code it is crucial to make sure that it does not introduce new leaks. The lesson we learned in the past is that even the tiniest leak can have devastating effects when it occurs for every connection Bro analyses. Whenever you are writing code for Bro's event engine, such as a new protocol analyzer, it's a good idea to double-check that all the memory it allocates will for sure be released later. As a rule of thumb: whenever you allocate memory but cannot guarantee that it will be freed at a well-defined point of time other than termination, there's likely something wrong.

There are various tools out there which can help with leak-checking. In the past we had most success with valgrind (good but slow with large input traces) and Google's perftools. Bro has some support built in for perftools which I'll summarize in the following. Note that perftools' leak-checking works on Linux systems only at the moment.

In the past, we have found quite a few leaks in Bro with perftools and we recommend to run new code through it to see whether there's something turning up. The nice thing about perftools is that its performance is sufficiently good that one can actually run Bro with some non-trival amount of traffic, which is crucial because often leaks are in code paths not triggered when analyzing just a few connections.

To use perftools for checking Bro for leaks, get the current source code from perftools' download page. Compile and install it with the usual ./configure && make install cycle.

Next, you need to configure Bro with debugging code and enable perftools support:

   > ./configure --enable-debug --enable-perftools 
   > make 

Make sure that the output at the end of the configure run indicates that configure has indeed found the perftools installation:

                    Bro Configuration Summary
    ==========================================================
      - Debugging enabled:      yes
      [...]
      - Using perftools:        yes

If it didn't find perftools, try giving it the paths to your perftools installation directly, as in the following example (replace <prefix> with whatever prefix you were using when configuring the perftools distribution):

 
  > export LDFLAGS=-L<prefix>/lib
  > export CFLAGS=-I<prefix>/include
  > export CPPFLAGS=-I<prefix>/include
  > export LD_LIBRARY_PATH=<prefix>/lib
  > ./configure --enable-debug --enable-perftools 

Once the configure output looks ok, compile Bro as usual with make.

The last preparation step is setting perftools' environment variable HEAPCHECK to local to activate perftool's memory checking:

 
  > export HEAPCHECK=local

Finally, start Bro as usual but add the option -m to the command-line, e.g,:

 
  > bro -m -r trace tcp

You likely want to run from a trace instead of live because the memory checking decreases Bro's performance significantly.

Once Bro terminates (which is fine to trigger via CTRL-C if necessary), perftools will output leak information into /tmp/bro.*.heap files. (There might be output saying that something "has failed" but it seems that you can you can safely ignore this.). The recorded data will only contain leaks which appeared within Bro's main packet processing loop; any leaks during initialization and termination are skipped as they won't cause any trouble during live operation.

At termination, Bro will print out a pre-built command-line to start the pprof utility. pprof is perftools' user interface to inspect the leak information, and you can just literally copy & paste the command line into your shell (except that you might want to skip the -gv option to suppress the graphical output in favor of the interactive interface)

Inside pprof, typing help shows the available commands. Most importantly, there's top which shows the top leaks as determined by perftools. See here for more information about how to interpret its output, and here for more details about perftools' heap checking in general.