bad timing

Previously, I had my fileserver sync time with my primary wireless router and some pool.ntp.org servers. Then I noticed this:

Jul  7 08:41:10 remote  /usr/sbin/ntpd[521]: skew change 255.181 exceeds limit
Jul  7 08:56:48 remote  /usr/sbin/ntpd[521]: skew change -34.707 exceeds limit
Jul  7 09:24:28 remote  /usr/sbin/ntpd[521]: adjusting local clock by -0.159868s
Jul  7 09:24:28 remote  /usr/sbin/ntpd[521]: skew change -32.121 exceeds limit
Jul  7 09:39:06 remote  /usr/sbin/ntpd[521]: adjusting local clock by 0.465410s
Jul  7 09:39:06 remote  /usr/sbin/ntpd[521]: skew change 132.634 exceeds limit
Jul  7 09:44:23 remote  /usr/sbin/ntpd[521]: adjusting local clock by 0.727679s

According to the documentation, there's no real-time clock on most OpenWrt compatible hardware. I guess when you consider the minimalistic setup of these devices, you can't expect everything.

I've now switched it around. My routers now sync with my fileserver and us.pool.ntp.org. My fileserver ignores the access points and syncs with:

server timex.cs.columbia.edu
server ntp-2.cso.uiuc.edu
server ntppub.tamu.edu
server ntp-1.vt.edu
server ntp3.cs.wisc.edu
server 0.pool.ntp.org
server 1.north-america.pool.ntp.org
server 2.us.pool.ntp.org

Those are the same NTP servers I use at work, with 3 pool.ntp.org servers thrown in. It just annoys me that OpenNTPD doesn't have the equivalent of "ntpq -p" to check the status of the NTP sync.

done! (mostly)

At this point, I've mostly finished my WRTSL54GS project. The only thing left is the DMZ subnet, but as I don't have a server to put in it yet, it's not a big deal.

As I'm an ubergeek, I've created a network diagram of the new setup. As I'm a lazy geek, the diagram is, of course, not 100% accurate. I actually allocated a physical port on each OpenWrt box to be in the wifi vlan (vlan3), which I've been using to attempt to penetrate my wireless network with a knoppix install & nmap. Next I'll try something else - nessus or something more "black hat".

I intended to keep more notes here, but like I said, I'm lazy. Here's the highlights -

• shfs absolutely rules. it's easy to setup and makes backups a snap. the most difficult part was trying to get the passwordless auth to work. I *believe* I ended up generating the keypair with ssh-keygen on my CentOS box, compiling dropbear on CentOS and following some notes on how to convert the OpenSSH keys to the dropbear format using dropbearconvert. However, I did it on my previous OpenWrt install and since the backups worked so well, I've been using the same keys ever since.
• I'm being absolutely draconian about the usage of the wifi subnet, so I've setup iptables rules to DROP all packets from the wifi subnet to the other private subnets. For some reason, packets kept flowing when adding the -j DROP rules to the FORWARD table, so I've added them to the INPUT table (input_rule table on OpenWrt) and that's done the trick. I *think* it might be because they're coming from the bridge interface (wifi is br0 - a bridge between eth2 and vlan3) and not a direct interface.
  
• I have to sing the praises of OpenVPN once again, as well as the OpenVPN GUI for Windows. Reliable, secure, flexible, simple - what more could one ask for? I made the VPN'd wifi subnet one number higher in the third octet of my lan subnet, so it's now a /23 in my hosts.allow, etc. etc. instead of the /24 it used to me. Simple, yet secure.
• Setting up the second OpenWrt box as a WDS repeater was pretty simple, as the instructions are good. The major tweaks were:
• Had to comment out portions of /etc/init.d/S05nvram, as it kept on putting back default variables I wanted un-set
• Disabled S35firewall, S50httpd, S50telnet and S60dnsmasq as the other OpenWrt box is doing the majority of the work
• Created an S35noipforward script with the contents being "echo 0 > /proc/sys/net/ipv4/ip_forward" as we don't want to do routing across vlans on the repeater here - let it shuffle the packets on to the primary one.
• And of course, the vlan setup for the WDS repeater:
  

  root@remote:~# nvram show | sort | grep ^vlan
  size: 1782 bytes (30986 left)
  vlan0hwname=et0
  vlan0ports=0t 2 3 4 5*
  vlan3hwname=et0
  vlan3ports=0t 1 5

  To be complete, here's the config for the primary router:

  root@lightsaber:~# nvram show | sort | grep ^vlan
  size: 3887 bytes (28881 left)
  vlan0hwname=et0
  vlan0ports=0 3t 5*
  vlan1hwname=et0
  vlan1ports=4 5
  vlan2hwname=et0
  vlan2ports=1 5
  vlan3hwname=et0
  vlan3ports=2 3t 5

  OpenWrt port 3 of the primary is connected to OpenWrt port 0 of the WDS repeater. vlan tagging is awesome.

100 hits and OpenWrt updates

I recently got my 100th hit on WheresGeorge.com. I'm an ubergeek for sticking with it, but hey - it's addictive.

I've also made progress on my OpenWrt install. Thanks to mbm's awesome switch/interface diagram, I've gotten all my VLANs setup and thrown my WRTSL54GS live with WhiteRussian 0.9. I've got my old WRT54G 2.2 mostly setup as well to be the switch/repeater in the other room, but I want to test my iptables rules that prevent the wifi network from accessing the local lan, except if you're using OpenVPN. The only reason for the trepidation is that they changed to iptables from a simple "-i $WAN" setup of setting the interface to a "-A prerouting_wan" chain and I want to be sure I don't foul it up.

P.S. I didn't end up using the flash memory card on my WRTSL54GS. I had to give the flash reader to my in-laws for their digital camera. Maybe one day I'll resurrect that end of the project for more storage there.

bah, roadrunner

I have a feeling TWC/RoadRunner botched a major upgrade to their network yesterday. My cable modem was offline for close to 12 hours. Two things were learned from this:

1) on OpenWrt, make sure "cache-file=/tmp/ez-ipup" is set in your /etc/ez-ipupdate.conf. A few days before the outage, my DHCP lease time dropped to 1 hour. Without ez-ipupdate caching my DynDNS account, I was temporarily banned. Also, my IP is now on a totally different subnet. Low lease time before outage + New ip after outage + High lease time after outage = they changed some serious shit. With a 12 hr downtime, they obviously thought it was going to go much better, but flubbed it.

2) The MS Update for KB911280 is incompatible with the current Road Runner dialer. I had to bang my head against a half-dozen TWC phone turds before one *finally* handed me off to tech support so I could get dialed in. The real dial-up tech support had this fixed in the time it took to reboot WindowsXP. Of course, it was too-little-too-late to be online for the early-morning maintenance work that I needed to do.

Anyway, I'm back online. It does make me think twice about using "crappy mega-corp" as my upstream, but it's the least of the evils available to me right now.

here be monsters

my thread on the OpenWrt forums explains the problems of late. CF issues have cropped up again, but only after i ran nvram set lan_ifname=eth0. i can boot off the internal flash and e2fsck the card and it checks out clean. and i see that when it boots, my red led lights showing that the card reader is seen and even the activity light flashes that something is going on, but all that i get from tcpdump is:

# tcpdump -vv -i eth1
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
22:24:39.486434 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.0.4919: UDP, length 1001
22:24:39.504939 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.255.4919: UDP, length 1001
22:25:05.436622 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.0.4919: UDP, length 1001
22:25:05.455131 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto 17, length: 1029) 192.168.1.1.1024 > 192.168.1.255.4919: UDP, length 1001

i.e. diddly for network. the first pair is when it boots off the internal flash, the second pair - i think it happens after it pivot_root's, but i'm not 100% sure yet.

interface layout & nvram cleanup

i'm trying to get the interface information for the WRTSL54GS straightened out so I can start setting up the DMZ. i found network config info in the wiki, including a diagram for my old WRT54Gv2.2, but not one for the new router. i'm in the middle of modifying the diagram to match the new router, but there's a lot of info - none of it too clear. i've posted on the openwrt forums asking for clarification. actually, i'm looking at making 2 diagrams - the "default" as shipped and my config - which will be w/o the bridge interface, with a dmz interface and a openvpn tunnel inteface setup.

in an effort to clarify things, i decided to tidy up my own setup by cleaning up the NVRAM variables (the safe way). so far, so good - after a reboot it's still there. :-)

root@OpenWrt:~# cd /tmp
root@OpenWrt:~# wget http://downloads.openwrt.org/people/kaloz/nvram-clean.sh
Connecting to downloads.openwrt.org[195.56.146.238]:80
nvram-clean.sh  100% |*************************************|  4702  00:00 ETA
root@OpenWrt:~# chmod a+x /tmp/nvram-clean.sh
root@OpenWrt:~# /tmp/nvram-clean.sh
Before: size: 11055 bytes (21713 left)
After: size: 3541 bytes (29227 left)
root@OpenWrt:~# nvram commit

dropbear and dsa keys

hm. i found this page on the openwrt wiki - DropbearPublicKeyAuthenticationHowto. since it's just me taking care of the router, i guess it's okay to auth to root or my non-root user using the same global authorized_keys file. but it just feels wrong. but dropbear has a really small memory footprint. argh, the problems of a sysadmin's conscience.

well, i've set it up on my active WRT54Gv2.2 router and i'll do it as a starting point on my WRTSL54GS next time i boot it.

... and we're back

well, i couldn't get it to boot off the CF anymore, so I just started over. i think the problem was that after i reloaded OpenWrt on the internal flash and installed the usb/storage drivers, it thought the CF filesystem "magically" changed to vfat and it didn't want to mount as root anymore. now i know that i mke2fs'd it. ah well.

so, i altered the swap technique by adding an /etc/fstab, making the swap init.d script S90swap and making it's entire contents "/usr/sbin/swapon -a". the contents of /etc/fstab are:

/dev/scsi/host0/bus0/target0/lun0/part1 / ext3 defaults 1 1
/dev/scsi/host0/bus0/target0/lun2/part1 swap swap defaults 0 0

i think my next step will be to dump dropbear and start running OpenSSH. i once spent entirely too many hours to get DSA keys to work with dropbear, unsuccessfully. i should have enough memory to run a real ssh daemon. besides, i think all the libs are pre-req's for OpenVPN anyway. or maybe remote syslogging. i've been seeing these weird scsi errors in 'dmesg' after it boots. i think it's for the empty slots in the flash memory reader - i believe i'm using sda and sdc (lun0 and lun2). i'm seeing errors for sdb and sdd as follows, but it's only the end of the error dump:

sdd : READ CAPACITY failed.
sdd : status = 1, message = 00, host = 0, driver = 08 
Current sd00:00: sns = 70  2
ASC=3a ASCQ= 0
Raw sense data:0x70 0x00 0x02 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x3a 0x00 0x00 0x00 0x00 0x00 
sdd : block size assumed to be 512 bytes, disk size 1GB.  
sdd: Write Protect is off
 /dev/scsi/host0/bus0/target0/lun3: I/O error: dev 08:30, sector 0
 I/O error: dev 08:30, sector 0
VFS: Disk change detected on device 08:30
sdd: Unit Not Ready, sense:
Current 00:00: sns = 70  2
ASC=3a ASCQ= 0
Raw sense data:0x70 0x00 0x02 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x3a 0x00 0x00 0x00 0x00 0x00

oops

hm. something went wrong - it won't complete the boot-up on the CF. i have a feeling it was the swap setup, as that init.d script was before the network init'd. no ping from the lan/bridge interface, the wifi interface doesn't come up (no light) and no dhcp on the wan interface. bleaugh.

luckily, that /sbin/init script from the OpenWrt has a test before it does the pivot_root, so I have it up and running with the internal flash. ahh, failsafes. fun fun fun. damn, this thing need a serial console.

paranoia

Since I don't have the wireless setup yet (specifically WPA and OpenVPN), I've created a failsafe to make sure people aren't peeking:

cd /etc/init.d
echo "ifconfig eth2 down" > S98nowifi; chmod +x S98nowifi

The wiki page on the WRTSL54GS let me know which interface is which.

creating swap

I found a 16MB smartmedia card laying around and figured it would be good for swap. Note that I had to install fdisk from the backports repository. I had previously partitioned the CF card when tooling around, so I didn't need it to mount the CF card as root, just e2fsprogs to reformat it. Anyway, on with the show:

# install fdisk
ipkg install fdisk
#find the partition
fdisk -l
# create the swap partition (partition 1, type 82)
fdisk /dev/scsi/host0/bus0/target0/lun2/disc
# install swap-utils
ipkg install swap-utils
# make it a swap partition
mkswap /dev/scsi/host0/bus0/target0/lun2/part1
# and do it
swapon /dev/scsi/host0/bus0/target0/lun2/part1
swapon -s
# make it persistent
echo "/usr/sbin/swapon /dev/scsi/host0/bus0/target0/lun2/part1" > /etc/init.d/S11swap
chmod 755 /etc/init.d/S11swap

Now I've got more virtual mem to do... something with.

root@OpenWrt:/etc# swapon -s
Filename                        Type            Size    Used    Priority
/dev/scsi/host0/bus0/target0/lun2/part1 partition       15984   0       -2
root@OpenWrt:/etc# cat /proc/meminfo
    total:    used:    free:  shared: buffers:  cached:
Mem:  31289344 12382208 18907136        0   770048  3854336
Swap: 16367616        0 16367616
MemTotal:        30556 kB
MemFree:         18464 kB
MemShared:           0 kB
Buffers:           752 kB
Cached:           3764 kB
SwapCached:          0 kB
Active:           2820 kB
Inactive:         1724 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:        30556 kB
LowFree:         18464 kB
SwapTotal:       15984 kB
SwapFree:        15984 kB

You can also read slightly less verbose instructions on the OpenWrt wiki.

getting the CF to be root

So, the UsbStorageHowto works pretty well. Basically, the steps I used were:

mount /dev/scsi/host0/bus0/target0/lun0/part1 /mnt
mkdir /tmp/root
mount -o bind /rom /tmp/root
cp /tmp/root/* /mnt -a
umount /tmp/root
umount /mnt 
cd /sbin
mv init init.old
vi init
chmod a+x init

and here's my version of init

#!/bin/sh
boot_dev="/dev/scsi/host0/bus0/target0/lun0/part1"
for module in usbcore usb-ohci scsi_mod sd_mod usb-storage jbd ext3; do {
        insmod $module
}; done
sleep 4s
mount "$boot_dev" /mnt
[ -x /mnt/sbin/init ] && {
        mount -o move /proc /mnt/proc &&         pivot_root /mnt /mnt/mnt && {
                mount -o move /mnt/dev /dev
                mount -o move /mnt/tmp /tmp
                mount -o move /mnt/jffs2 /jffs2 2>&-
                mount -o move /mnt/sys /sys 2>&-
        }
}
exec /bin/busybox init

Of course, what's not mentioned is that your system is reset to as if you just installed OpenWrt. So you have to set the root pw, update ipkg.conf, install pkgs, etc. etc. as per my last blog post.

BTW, after it's all said and done, the filesystems show up as follows:

root@OpenWrt:~# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/root                 1024      1024         0 100% /mnt/rom
/dev/mtdblock/4           6272      1304      4968  21% /mnt
none                     15276        36     15240   0% /tmp
/dev/scsi/host0/bus0/target0/lun0/part1   121203      8583    106362   7% /mnt/mnt/disc0_1
/dev/scsi/host0/bus0/target0/lun0/part1   121203      8583    106362   7% /
root@OpenWrt:~#

With 32MB of RAM and 128MB of disk... it's 1990 all over again!

Tony

getting things started

Well, I got off my ass today and started working on getting my WRTSL54GS up and running. So far, the notes I had made previously work like a charm for getting OpenWrt up and mounting the CF card. Next is to get it mounting the CF card as root.

tftp 192.168.1.1
binary
rexmt 1
timeout 60
trace
put openwrt-WR1.0rc5-wrtsl54gs-squashfs.bin

telnet 192.168.1.1
passwd
# logout/login
ssh root@192.168.1.1
wifi down
ifdown wifi
ifconfig eth2 down

Then with some help from the OpenWrt wiki.

cd /etc
rm ipkg.conf
cp /rom/etc/ipkg.conf ipkg.conf
vi /etc/ipkg.conf
#src backports http://downloads.openwrt.org/backports/rc5
ipkg update
ipkg install kmod-usb-core
ipkg install kmod-usb-ohci
ipkg install kmod-usb-storage
ipkg install kmod-usb2
ipkg install kmod-vfat
ipkg install kmod-ext3
ipkg install e2fsprogs
#reboot
fdisk /dev/scsi/host0/bus0/target0/lun0/part1
#reboot
ln -s /proc/mounts /etc/mtab
mke2fs -j /dev/scsi/host0/bus0/target0/lun0/part1