Things I Wish I’d Known

This set of pages is where I keep information that wasn’t obvious to me when I went out to solve a problem. If you know something nifty that wasn’t easy to learn, let me know and I’ll be happy to put it up on this site. (You can keep the copyright on your submission, if you wish— just let me know how to annotate it.)

Developing and Debugging

My friend David Priebe has just begun studying C++, and it’s been interesting remembering the various principles that I learned years ago, then internalized and lost from conscious memory. Some of his insights, and my commentary:

When you’re debugging, it helps to check the line before the one where the compiler reports an error.
And bear in mind that the line before the one reported can be in an entirely different file, depending on the order of #includes! I don’t know if this is a problem with modern C preprocessors, but I’ve sometimes had cross-platform problems where someone checked in a header file protected with #ifndef/#define/#endif and no newline at the end. This worked fine on Windows, but made a Solaris-based compiler plotz.
In general, I start fixing reported errors from the top of the file heading downward. Files should be short enough that you can recompile quickly enough that you can make a change and recompile quickly to make sure it worked.

Important Principles

One important principle to keep in mind when doing software engineering: externalize as much knowledge as possible into your code. This can be done through the code structure, the comments, or both. Never assume that the person reading the code knows everything that’s going on. That person could be a new hire next week or next year, or just yourself in six months revisiting code that you’ve forgotten since then.

I’m in the habit of commenting all my C/C++ #includes with mentions of what I’m including them for (which comes in handy later when you’re splitting the file up into pieces after it grew to a size where it’s worth breaking up and when turning features on and off during the debugging process). If I’m implementing from an RFC, I tend to include liberal quotations from the RFC as comments in the code.

It takes only a little extra time to do this at the start, and makes program maintenance much easier down the road.

System Calls

Memory Mapped I/O

One of the nifty things I discovered early in my career was a Unix system call named mmap(). It’s a very nifty call for systems that have virtual memory: instead of taking the trouble to allocate a big buffer in memory (which will be swapped out to disk) and then reading a whole file into that buffer, it just directly maps a file on disk into memory. (Very handy if you're processing gigantic data sets.) I once wrote a C++ streambuf that used mmap() to access a file for reading, though I never did get around to doing the writing portion of it. The Windows equivalent (discussed in the MSDN article Managing Memory-Mapped Files in Win32) uses the calls CreateFileMapping(), OpenFileMapping(), MapViewOfFile(), MapViewOfFileEx(), UnmapViewOfFile(), FlushViewOfFile(), and CloseHandle().

Binaries

It helps to understand just what’s going on in an executable so you can debug problems— a recent one I ran into is that under gcc on Linux, you only need to set the -fPIC flag (for compiling position-independent code) on the binaries you use in a shared library; under the Intel C++ compiler, you need to set -KPIC when you compile a regular executable that makes use of a shared library! (This won’t harm you if the library doesn’t have any routines that get called when it’s loaded, but you can run into trouble if you have static constructors that make use of their own this pointer...)

On Linux, handy commands are fuser (for figuring out who’s using that file), strace (to eavesdrop on system calls), ldd (to see which libraries the program is loading), objdump (to peek into object files), size (discover the section sizes of a binary with -A) and strip (to remove the symbol table before shipping). On Win32, get regmon and filemon and use dumpbin (like ldd and nm) and depends (a very thorough ldd for Win32’s amazing webwork of DLLs).

C++

The Guru of the Week articles on comp.lang.c++.moderated are good brain twisters.

Iostreams

Iostreams can be confusing to navigate, but they’re worth taking the time to understand if you need to do things like make a web server plug-in that has to talk to a C API. The actual thing you want to mess with when hooking up a new form of I/O is not an iostream, but the streambuf that the iostream uses. Streambufs have a buffer of data and a bunch of pointers that show where data is coming in and out and getting read and written, and they aren’t easy to get right. When I had nothing but Solaris man pages and MS Visual Studio to work with, I ended up having to look at the source code in the Microsoft C Runtime to figure out how the really important calls (sync(), overflow(), underflow()) worked; you can also browse the GNU Standard C++ Library source. The end result, though, was being able to freely use iostreams to pump data through NSAPI and ISAPI and Apache without any of the code having to understand the different idiosyncracies of the platforms, so it was well worthwhile.

STL: the Standard Template Library

The STL takes some getting used to, but it’s very effective. When you have a reliable class library handling all the allocation and deallocation for your data structures, a lot of headaches go away. When I was doing a cross-platform file class, I had a vector of auto_ptrs to the files (since some were actually the directory subclass of the file). (Michiel Salters informs me that this is no longer possible with the latest STL; vector elements must be CopyConstructible, so a helper class was introduced to prevent vectors of auto_ptrs from compiling. Using a shared_ptr from the Boost library in place of the standard auto_ptr should do the trick.)

Memory Alignment

One interesting feature of many processors is that they get unhappy if you try to read a fundamental data type, like a 4-byte integer, that isn’t aligned to a 4-byte boundary. This sort of problem is usually taken care of by the compiler, but when you get involved in networking code and you're pointing a structure pointer into an arbitrary spot in a buffer, you can get interesting crashes. Some compilers can be persuaded to read misaligned data. If you’re using C++, you can declare your class as normal and have the accessors all copy the data into a variable declared on the stack, then return the variable on the stack.

Networking

Non-Blocking Sockets

Normally when you read data off a TCP socket, you just give a buffer size and a file descriptor and make your system call, and eventually it comes back when the buffer is full or the connection is closed from the far end.

I once ended up having to cope with a system where data of an unknown length was being pumped over the network by a PC that was decoding stock ticker information that came in off a satellite dish. The guy who had written the library to do this had solved the problem by doing a whole lot of read(1) calls— yes, that’s a system call for every single byte coming down the connection. That’s a huge wasteage of CPU.

I did some research on the Internet and found the UNIX Sockets FAQ, where I learned about putting sockets in non-blocking mode, which you by using fcntl() to set the O_NONBLOCK flag on the socket (and don’t forget to do it on the flags that you retrieved via fcntl() first, or you could put the socket in a weird state!). When you do that, read() returns instantly with whatever amount of data is available from the socket. If you need a timeout, use select().

If you don’t want to go into non-blocking mode, you can also use ioctl() with the FIONREAD parameter to find out how many bytes are currently unread on your file descriptor.

Other useful FAQs are the Winsock Programmer’s FAQ, Winsock Development Information, and the Raw IP Networking FAQ.

Idiosyncracies

Microsoft’s source control integration between Visual Studio 5 and SourceSafe was pretty unreliable. At CoVia, I often had to go into the files marked DO NOT EDIT and manually put in the SourceSafe information so new projects would integrate with the database. It required editing both the workspace file and the project file, and I never did figure out what all the flags were in there. Once you’ve got a good one to copy from, you can keep repeating the pattern; you just can’t trust the software to get it right for you.

Cross-platform work can be a headache, and I recommend you do it by engineering a good class library to hide all the differences between platforms from the start and proceeding from there. (At CoVia, I had a File class that had an STL vector of Rogue Wave RWCStrings and understood how do to path arithmetic and the different path formats (UNIX, Windows, UNC, ...) and a Directory subclass that made directory navigation easy. Hint: save on stat() calls by using a mutable structure to store the information in the class.

Under VxWorks, I found that Windows NT and Windows 98 have different ways they treat output redirection. This make rule worked fine on NT:

%.s : %.html
	sed -e "s/ *</</g" -e "s/> */>/g" $< > $*.tmp
	binToAsm $*.tmp | sed -e "s#binArray#$(*F)#" > $@
	rm $*.tmp

but plotzed on 98. When I changed it to:

%.s : %.html
	sed -e "s/ *</</g" -e "s/> */>/g" $< > $(subst /,\,$*.tmp)
	binToAsm $*.tmp | sed -e "s#binArray#$(*F)#" > $(subst /,\,$@)
	rm $*.tmp

everything worked fine, though I would’ve had to make conditional rules if this were also building on Solaris as well.

I also found that you can’t manipulate multiple rules like that in a single project in VxWorks’ Tornado. Each rule is filed under the target, so if you have one rule for %.s: %.html and another for %.s: %.jpg, you have to add them by hand-hacking the WPJ file and never, ever editing the rule through the GUI again.

Nifty Tools

Okay, I admit it: I’m a GNU Emacs addict. Yes, the editor takes up huge quantities of system resources and contains its own implementation of LISP. Yes, it requires that you learn a whole bunch of keyboard mnemonics (though the mouse integration is pretty good these days). But it’s a windowing editor that doesn’t force me to cope with a deranged multiple-document interface, it has superb macro and regular expression facilities, and it integrates quite well with CVS and C/C++ compilers during development... as long as you aren’t depending on the GUI. (Though in server mode, you can hook up an emacs client to your GUI so doubleclicking on an error line will bring up the file in Emacs.) The GNU Emacs FAQ for Win32 will tell you how to get it working on Windows, and you can even integrate it with Visual Studio.

GNU make is an excellent tool. I’ve been able to put together some pretty effective build environments using GNU make— I whipped up the one for CoVia from scratch and got it to build on Solaris and Windows. VxWorks also uses it, though on there you have to know how it uses TCL to convert its WPJ files to GNU Makefiles.

I prefer CVS to SourceSafe, for a multitude of reasons, including the fact that SourceSafe has this interesting tendency to corrupt its own database. The modern versions of SourceSafe will let you un-corrupt the database without forcing everyone on the network to log off (most easily performed at the circuit breaker box), but you still have to know to run it every week. By the time the corruption shows, it’s usually already too late. CVS works from the command line, as does SourceSafe, and the CvsGui project is doing quite well. I’m curious as to how well Subversion is working out, and looking forward to Katie (a cross between CVS and NFS) maturing.

Rogue Wave makes some excellent class libraries. Threads.h++ makes threading issues easy to deal with, with guard classes that automatically lock resources at construction time and unlock them as their destructors are called, and I still can’t decide which is more cool: the recursive readers-writer lock or the thread pool that has a queue of functors you can throw into it for processing as threads become available. Tools.h++ is most memorable for its nifty string, date, time, and regular expression classes, as a lot of its other goodies are replicated by the STL. Tools.h++ Professional makes networking very easy— I was able to put together a multithreaded web server very quickly using the classes there. DBtools.h++ has one of the best uses I’ve seen for operator overloading: its classes are designed so you can perform SQL queries using C++ syntax. (The [] operator on a table object gives you a column object reference, operators between columns and other forms of data yield expression objects, and expression objects can be fed to selectors.)

There’s a delightful UNIX utility that lets you watch all the system calls being made. On Solaris, it’s called truss; on Linux, it’s strace. It makes it much easier to see what files and libraries are being accessed and what’s going wrong with a program for which you lack source code. Win32 isn’t quite so helpful, but there are some nifty utilities at Sysinternals Freeware like Filemon, which lets you watch filesystem activity, and Regmon, which lets you watch registry activity. Sysinternals is full of useful information for Win32 developers; any time you have to deal with weird low-level Windows behavior, it’s worth looking the site over.

Cygwin provides Unix utilities that make Win32 a lot more livable; if you use tcsh, you can avoid the problem with command line editing screwing up in insert mode by putting settc im '' and settc ei '' into your .cshrc. Cygwin comes with Perl, but ActivePerl is also available, and effective, without Cygwin.

System Administration

Sendmail

Sendmail is a very complicated piece of work. Fortunately, there are packages to make it easier to customize. Red Hat’s Sendmail HOWTO has useful information about how to use sendmail-cf to convert readable files into a hard-to-read sendmail.cf configuration.

If you lose the admin password for your NT or Win2K box, there is an Offline NT Password and Registry Editor, and countermeasures [PDF] to this vulnerability. There’s a nice NSA-commissioned study of Windows NT Security Guidelines at Trusted Systems Services.