I never remember the syntax for this, should you need to build a port without X11 support do this:

# make WITHOUT_X11=yes