Dave Naffziger's Blog

I’ve finally run a Boston qualifying marathon! Despite running 14 prior marathons, I don’t think I’ve ever properly trained for one.

I completed the Seattle Ghost Marathon Thanksgiving day weekend in 3:07:59.

Rahul and I ran the first 14 or so miles together, but his recurring Achilles injury flared up and he wisely decided to stop around mile 20. We’re planning on another joint attempt on Feb 19 (http://birchbaymarathon.com/)

I’ve been able to run half-marathons pretty fast (ran my fastest a few weeks prior: 1:29:23), but have generally faded after the first 15-18 miles. I faded in this run also, but the fade was much less pronounced - go figure training works.
The event itself was great. With only ~200 runners running a mix of halfs, fulls and ultras, and starting at a mix of times (they allowed non-qualifying runners to start early), there always seemed to be a runner in sight. The course is a nice flat out and back along Lake Washington (actually, a double out and back) and only rarely separated from the water.

One runner attempted an Olympic qualifying time, and unfortunately just missed it by a couple of minutes. This added to the general excitement around the event - word would pass among the other runners about whether she was on pace, etc.

I loved having my family there for the run though. My sister and her husband were awesome supporters before and during the race, making sure I was eating pasta and drinking water the night before and banging pots and pans for the runners. I think this was also the first time my dad saw me run which made it extra fun for me.

Sophie is also just beginning to understand the concept. When I would go for training runs, she would say ‘Daddy is going to run. With Uncle Rahul’. And when I returned she’d say ‘Daddy is all sweaty. He needs to shower’. My sister made Sophie a sign and she cheered for Rahul and I. Later in the day she recounted the morning:

I cheered for the runners! I said ‘Go Daddy Go’ and I said ‘Go Uncle Rahul Go’. And I held a sign.

Now on to hernia surgery in late December =P.

I’ve received a bunch of questions from friends starting companies recently about the tools that I’d recommend for running a business. Hopefully this will help others in the future.

My general bias is for web-based solutions for everything admin, so I never considered desktop solutions. Once you have a finance department at your company your needs will be different. My objective is to build our business as large as I can without needing a finance department (or person). SAAS-only solutions make it a ton easier to engage accountants, bookkeepers, etc. helping further delay the need for a finance person (overhead).

Paycycle (now Intuit Payroll) for payroll
I can’t recommend Paycycle enough. In fact I would make any this service the central component of your accounting decisions. While I was initially lukewarm on the service, I have rapidly grown to consider it essential.

Maybe ADP compares, I haven’t looked at it and their pricing isn’t on the web. But, I’ve rarely seen an offline company offer a better online solution than an online-only company.

Few first-time entrepreneurs really understand the web of tax and payroll issues that they need to deal with when starting a company. There are monthly payroll filings and tax payments to the IRS. There are 2-3 quarterly payments likely required by the state, often to different agencies (payroll tax, unemployment tax and maybe workers compensation). And there is a whole slate of annual payments.

Paycycle files and pays nearly all of those taxes. If they can’t pay and file, the calculate the numbers for you. Deductions, vacation, bonuses, holidays, commissions … handled.

Expensify for expense reports
While a corporate business card is the single best way to handle all expenses at a super-early stage company, invariably you end up needing to pay cash for cab rides, etc. Once you begin adding employees, you’ll need to handle their expenses as well.

Expense reports suck and Expensify makes them way more streamlined. Creation, submission, approval and payment of reports is super-easy, and smart phone apps let you easily upload photos of receipts. Not to mention the benefit of having digital records of all of this.

On the downside, the Expensify UI can be a little clunky. While it is way better than most business services I’ve used, it feels disjointed at times and I’d like to see them focus more on raw usability. Exporting the data isn’t as smooth as I’d like, but my gut tells me we can improve on our setup (and writing this tells me that I should make sure we investigate that ASAP). With that being said, I haven’t found a better solution.

QuickBooks Online for accounting
Sigh. QuickBooks is the swiss army knife of accounting solutions. It can just as easily support construction businesses, retail storefronts, condo associations and your startup. But it is really heavy, bulky and slow. The web version attempts to emulate a desktop app, and breaks nearly all web usability guidelines. I find it takes me way too long to pull out standardized reports, as well as do things that feel that they should be simple.

Still, a stunning 71% of startups use QuickBooks according to this BestVendor infographic. That is a higher market share than every other service including stalwarts like Salesforce, Google Apps, Google Analytics, etc. That 71% is compared to 5% who use Excel and 4% who use Xero.

Why?

Every third party financial service integrates with Quickbooks. One of the most time consuming aspects of accounting is getting data into the system so integration with every other service you use is critical. Even new services like Xero try to integrate with files exported specifically for QuickBooks (and do a poor job at it).

Not to mention that many of the competitors have lame limitations that severely impact their utility. For example, Freshbooks won’t email invoices (they’ll email links to invoices, but not the actual invoice).

We’re starting to reach the limits of what QuickBooks Online can do (deal with foreign currencies), and I’m definitely worried about the additional overhead we will inherit when we switch providers.

EarthClassMail for handling checks
If you are selling to enterprises, you will likely have to accept payments by check. Dealing with checks is another hassle that I’d prefer not to have to deal with. Among other annoyances, it requires the mailing and/or depositing of checks along with a paper trail to record the transaction history. Not to mention the hassle if they just get lost along the way.

EarthClassMail is a great solution to this problem. They provide a lockbox-like service that automates the receipt, scanning and deposit of checks. Just provide your customers a different billing address and checks hit your bank account faster than you could manually do yourself. The scans of checks and their related payment information also creates a perfect digital paper trail.

Nearly all banks provide lockbox services, but my experience with the ‘analysis checking’ services provided by banks has been universally miserable - they just have terrible UI. I can’t speak for all banks, so maybe there is a gem out there.

What solutions do you use? I’d love to improve on what we have.

My copy of Venture Deals arrived a little while back, but it wasn’t until my trip to SF this weekend that I had a chance to read it. My plan was to skim the book and then pass it on to one of my friends that is actively raising a round. Instead, I’m keeping it on my bookshelf next to the essential Entrepreneur’s Guide to Business Law and sending out a few copies.

I found myself alternately skimming sections and reading every single word. The book is succinct and doesn’t unnecessarily repeat itself. I was able to quickly determine whether a particular section had something new to teach me and dig in as needed. The book easily fit into a few uninterrupted hours on a flight, but I expect I’ll reference it from time to time.

As an early entrepreneur, a disproportionate share of your legal counsel’s time is effectively spent educating you. No sane entrepreneur should negotiate an obscure point on a term sheet that he doesn’t understand. In my experience, one of the reasons legal bills are often higher on a first-time entrepreneur’s company is this education curve. Read the book. The last thing you need is to shovel money from your completed financing to the lawyer that had to coach you through terms you could have quickly taught yourself.

But, the value of the book isn’t limited entirely to negotiating a venture round. I also found the tips on negotiation to be both timeless and more broadly applicable (I suppose there are negotiation books for that also). For example, most corporate legal processes are set up to exploit the tendencies of smaller companies. They take forever to process revisions and can involve many back-and-forth discussions. While I’m not particularly sensitive to the length of the process, I’m extremely sensitive to the time I spend on the process. Looking back, I see myself consistently agreeing to slightly worse terms the more time I spend on the negotiation. This wasn’t something I was aware of, and that self-reflection will be useful going forward.

Some of the biggest gaps in the book are easily addressed if you read it while online - I find that spreadsheets help me internalize the dynamics of financing terms much better than printed text, and there are plenty of resources for that online.

If you are operating (or hope to operate) a startup, Venture Deals is a great asset even if you don’t plan on raising any investments.

I received this awesome voicemail on my BrandVerity Google Voice number:

The Google Voice transcription wasn’t usable, so I did a quick transcription myself. You do really have to listen it to appreciate it though:

Hi this is Roberts, Data Service Provider for Google. I wanted to give you guys a call, we’ve got a pretty good opportunity in your marketplace. We’ve got some positions that opened up on the search engines Google, Yahoo and Bing and wanted to see if there’s any interest as far as improving your ranking on the search engines. When you get a couple of minutes give me a call. Phone number here is 800 219 6018…

I’ve fielded plenty of snakeoil SEO pitches before, but this one receives particularly high marks.

“It turns out the cloud is actually just some place in Virginia” — Tweet by jckhewitt

We were impacted by the long EBS outage at Amazon’s US-East-1 data center. BrandVerity uses a geographically distributed collection network to find and store ads and the sites they send traffic to for our customers. While less than a third of our servers are hosted by Amazon, we were significantly affected by the downtime because we use Amazon’s Elastic Block Store (EBS) to store some of our data. In total it was 28 hours before Amazon restored our EBS volumes. However, we were able to run a number of our systems during the outage by employing traditional disaster recovery techniques (restoring from backups, etc.).

We had a few expected and unexpected observations and lessons learned from the outage.

Unexpected Dependencies and Disappearing Redundancies
Many companies launched over the last few years run on AWS (over a third of Y-combinator startups have their principal domain at AWS Source). Supporting technologies that are especially useful to SAAS companies are disproportionately run on AWS.

We found unexpected dependencies in our technology vendors that we assumed would be independent of our own issues. An outside monitoring vendor had an AWS dependency that took down their monitoring. Not only was our own in-house monitoring impacted, but our backup monitoring went down completely.

Of course, the bigger issue is that Amazon’s Availability Zones weren’t as independent as Amazon has advertised. There is much discussion of this on the web at the moment and Amazon’s post-mortem provides an excellent view into the issue.

The outage has certainly raised the visibility of true multi-datacenter redundancy, which had been replaced in many organizations by multi-availability zone redundancy.

Cloud-based Outages Allow Teams to Focus on Software-based Recovery Options
Unplanned outages happen. In our prior companies, hardware outages have resulted in scrambles at the data center. Since the hardware issues would be the most immediate, nearly all technical personnel were on hand helping the systems administrators.

In this outage, we were a little uncomfortable to be a step removed from the core issues, but it allowed us to get back up and running more quickly.

Our team was able to focus on software-based recovery options. Rather than racking and unracking hardware, installing disks and operating systems, we were able to instead focus on multiple recovery options. We pursued two independent paths for recovery and were in a great position should our primary recovery option have run into unexpected problems.

Using Amazon AWS is (was?) Today’s Equivalent of “buying IBM”
When we selected Amazon’s cloud platform several years ago it hadn’t become the standard bearer it is today.
We were pleasantly surprised by our customer conversations the morning of the outage. Customers readily understood the source of the problems and very few of them attributed the issues directly to us. While we are fully responsible to our customers, they generally afforded us a wider margin and assigned more blame to Amazon.

I’ve had to do a few customer calls in the past where there wasn’t a third party involved in an outage. Even when the issue was an unlikely and unfortunate collision of events, they seemed to be generally unhappier with us.

Customers are certainly more likely to understand that Amazon was having a massive outage than the low probability nature of a near simultaneous failure of two disks in a Raid 5 volume.

We didn’t expect this response, but certainly didn’t mind it. It will be interesting to see if this perception will hold given this latest outage.

What have your lessons and observations been from the outage?

As I’ve gotten older, the event that has consistently brought friends together has been weddings. As my friends have settled the weddings have grown less frequent.

With one exception. Every year for the last seven, a good friend has organized a trip to Park City. A dozen college friends made the trip each year. Not everyone came every year, but everyone came every few years.

This will be the first year that I haven’t made the trip.

I’ve just learned about the only other event that consistently gets old friends together - funerals. Scott Cook, a friend from I think fifth grade just passed away. I’ve only seen him sporadically since high school. Last time was a few years back at Sebastian’s wedding.

As my college friends have gotten older, the annual trip to Park City has gotten harder. I suppose it is fitting that the one-time I can’t make the trip I’ve never been so reminded of its value.

I’m excited about seeing my high school friends again, but I wish it was under better circumstances.

Last night around 11:30 I received an email from a friend that my Facebook account had spammed him.

On logging, I learned that my account had:

• Created an event: ‘No Fees PS3 Slim for x-mas’
• Invited all my friends to attend the event
• Posted that I was attending the event to my Facebook status

Here is the screenshot of the ‘event’:

Awesome.

It looks like the email went out at 11:05 PM. My friend emailed me at 11:28 PM and I had the event down at 11:42 PM. But there is no pulling back the emails to my friends.

Figuring out how it happened

My first thoughts were that it was a CSRF attack or related to the Gawker hack (I had a barely used account there).

First step was to visit the link using a sandboxed browser. Here were the redirects (I’m skipping a few that were just internal to the sites in the path):

• http://therently.info/
• http://174.139.89.156/ps
• http://www.affiliatecashpile.net/aff_c?offer_id=547&aff_id=2836
• http://my.amazingfreerewards.com/DefaultPage.aspx?nm=01pgdfz2uqxr&s=acp2836!547

Nothing of much interest in the redirects. It was notable that I didn’t record any requests to Facebook. The nature of this attack was nearly identical to a recent Facebook worm (Sep 2010) that exploited a Cross-Site Request Forgery (CSRF - an attack that simulates requests to a third party site to create some action on that site). Facebook also seems to have consistent problems with CSRF attacks.

Still, the lack of a CSRF request in my one example doesn’t guarantee that it wasn’t a CSRF attack. All of my browsing at the time of the exploit was on very safe sites. CSRF continued to look unlikely.

Weak passwords are lame. Using duplicate passwords is lamer.
I use several low-security passwords on accounts that I generally don’t care about. Gawker’s system was one of them. Most of these accounts I use only a handful of times so I haven’t really been concerned about access to them.

Unfortunately, when I originally signed up for Facebook, I used the same low security password as well. While I quickly changed my facebook password, I found it very difficult verifying that the password was the source of the exploit.

Nothing else in the account had been changed (more indicative of the CSRF attack).

Gmail has this awesome feature that allows you to see which IPs have logged into your account. It is a great way to ensure the integrity of your account. Of course Facebook makes no such data available (and yes it is still uses unencrypted cookies and is vulnerable to Firesheep).

The password was a mostly random string of letters and numbers. However, the fundamental flaw was that it was only 6 characters. That leaves the password vulnerable to a brute force attack.

How vulnerable?
How Secure is My Password estimates it would take 3 minutes for a desktop PC to ‘crack’ a similar password.

This Slate tool tells you if your email was in the hacked Gawker database. It also tells you if your password has already been cracked. Here was the message it gave me:

Your password was released, and it’s been decrypted. You should change it ASAP.

How was my password cracked?

Gawker states that they store passwords in an encrypted form in their database. The method used by my sites is to store a hash of the password rather than the actual password. The hash is usually a one-way function, so to log you in the site has to generate a hash of the password you entered and then compare it to the stored value in the database.

For example, if my password was ‘davenaff’, the md5 hash would be: 7c074125deb947e96fc7bb8de60c6e17. For a hacker to reverse my password, they would need to generate all possible combinations until they found a hash that matched my password.

There are databases and websites that have pre-calculated a dictionary of md5 hashes to their original values. (These have a cool name - ‘Rainbow Tables’). To counter this approach, many websites use salted hashes. Now, to crack the hash you need to know the salt, but since Gawker source code was hacked the hackers also have the salt (if one were used).

The hackers reportedly obtained 500k emails and as of this morning had decrypted 185K of them. Presumably they (or others) will continue to decrypt more as they build an increasingly larger dictionary of hashes to test against the remaining 315k passwords.

I assume that they at least ran a ‘rainbow table’ attack against all dictionary words and 6-digit passwords. The computation complexity begins to increase substantially as you add characters (from the howsecureismypassword site):

6 chars	3 mins
7 chars	2 hrs
8 chars	3 days
9 chars	117 days
10 chars	11 years
11 chars	417 years

And those stats are just for a desktop PC. Purpose built hardware and algorithms would go much faster.

Anyway, since my password was outted, I’ve started seeing warnings (or messages that could be considered warnings) from people trying to access some of my accounts. For example I got this message from linkedin earlier this evening:

Dear Dave Naffziger,

In order to ensure that you continue to have the best experience using LinkedIn, we are constantly monitoring our site to make sure your account information is safe.

We have recently disabled your account for security reasons. To reset your password, follow these quick steps:

Go to the LinkedIn website
Click on “Sign In”
Click on “Forgot Password?” and follow the directions on the website

Thank you,
The LinkedIn Team

Sweet, I wonder how many of those messages I’ll get…

The thing is, I know better than this. But, I guess it was time to relearn a few lessons:

• Regularly consider the password I’m using when I log into sites. Upgrade when necessary
• In all cases, use a password with at least 9 characters. My high security passwords have many more, but I’m certainly willing to type a few extra keystrokes if it means avoiding spamming my friends
• Respond quickly. The lamest thing about this is that I read about the Gawker hack before my facebook account was compromised. AND, I recognized that I probably had a Gawker account AND that the password was the same as my FB account. I literally had strung together all of these thoughts and didn’t take action until my account was compromised.

I think I’ve got all of the formerly ‘low security’ accounts I care about changed. Now, if I can just figure out how to change the password on evite…

From this fascinating CNET article on a few of the techniques that the Russian spies used to exchange data:

…the steganographic program was activated by pressing control-alt-E and then typing in a 27-character password, which the FBI found written down on a piece of paper during one of its searches.

While online passwords never need to be this complex - centralized management can prevent brute-force attack. Arguably, longer passwords are more important for local software (that can be stolen, then brute-forced). In fact, the leading theories on the ‘cracking’ of the wikileaks video suggest that they brute-forced the password that unlocked the encrypted contents.

A 27 character password certainly makes brute-forcing the password impossible. But human nature, even to extremely well-trained spies is to write things like this down.