Dave Naffziger's Blog

I’ve finally run a Boston qualifying marathon! Despite running 14 prior marathons, I don’t think I’ve ever properly trained for one.

I completed the Seattle Ghost Marathon Thanksgiving day weekend in 3:07:59.

Rahul and I ran the first 14 or so miles together, but his recurring Achilles injury flared up and he wisely decided to stop around mile 20. We’re planning on another joint attempt on Feb 19 (http://birchbaymarathon.com/)

I’ve been able to run half-marathons pretty fast (ran my fastest a few weeks prior: 1:29:23), but have generally faded after the first 15-18 miles. I faded in this run also, but the fade was much less pronounced - go figure training works.
The event itself was great. With only ~200 runners running a mix of halfs, fulls and ultras, and starting at a mix of times (they allowed non-qualifying runners to start early), there always seemed to be a runner in sight. The course is a nice flat out and back along Lake Washington (actually, a double out and back) and only rarely separated from the water.

One runner attempted an Olympic qualifying time, and unfortunately just missed it by a couple of minutes. This added to the general excitement around the event - word would pass among the other runners about whether she was on pace, etc.

I loved having my family there for the run though. My sister and her husband were awesome supporters before and during the race, making sure I was eating pasta and drinking water the night before and banging pots and pans for the runners. I think this was also the first time my dad saw me run which made it extra fun for me.

Sophie is also just beginning to understand the concept. When I would go for training runs, she would say ‘Daddy is going to run. With Uncle Rahul’. And when I returned she’d say ‘Daddy is all sweaty. He needs to shower’. My sister made Sophie a sign and she cheered for Rahul and I. Later in the day she recounted the morning:

I cheered for the runners! I said ‘Go Daddy Go’ and I said ‘Go Uncle Rahul Go’. And I held a sign.

Now on to hernia surgery in late December =P.

I’ve received a bunch of questions from friends starting companies recently about the tools that I’d recommend for running a business. Hopefully this will help others in the future.

My general bias is for web-based solutions for everything admin, so I never considered desktop solutions. Once you have a finance department at your company your needs will be different. My objective is to build our business as large as I can without needing a finance department (or person). SAAS-only solutions make it a ton easier to engage accountants, bookkeepers, etc. helping further delay the need for a finance person (overhead).

Paycycle (now Intuit Payroll) for payroll
I can’t recommend Paycycle enough. In fact I would make any this service the central component of your accounting decisions. While I was initially lukewarm on the service, I have rapidly grown to consider it essential.

Maybe ADP compares, I haven’t looked at it and their pricing isn’t on the web. But, I’ve rarely seen an offline company offer a better online solution than an online-only company.

Few first-time entrepreneurs really understand the web of tax and payroll issues that they need to deal with when starting a company. There are monthly payroll filings and tax payments to the IRS. There are 2-3 quarterly payments likely required by the state, often to different agencies (payroll tax, unemployment tax and maybe workers compensation). And there is a whole slate of annual payments.

Paycycle files and pays nearly all of those taxes. If they can’t pay and file, the calculate the numbers for you. Deductions, vacation, bonuses, holidays, commissions … handled.

Expensify for expense reports
While a corporate business card is the single best way to handle all expenses at a super-early stage company, invariably you end up needing to pay cash for cab rides, etc. Once you begin adding employees, you’ll need to handle their expenses as well.

Expense reports suck and Expensify makes them way more streamlined. Creation, submission, approval and payment of reports is super-easy, and smart phone apps let you easily upload photos of receipts. Not to mention the benefit of having digital records of all of this.

On the downside, the Expensify UI can be a little clunky. While it is way better than most business services I’ve used, it feels disjointed at times and I’d like to see them focus more on raw usability. Exporting the data isn’t as smooth as I’d like, but my gut tells me we can improve on our setup (and writing this tells me that I should make sure we investigate that ASAP). With that being said, I haven’t found a better solution.

QuickBooks Online for accounting
Sigh. QuickBooks is the swiss army knife of accounting solutions. It can just as easily support construction businesses, retail storefronts, condo associations and your startup. But it is really heavy, bulky and slow. The web version attempts to emulate a desktop app, and breaks nearly all web usability guidelines. I find it takes me way too long to pull out standardized reports, as well as do things that feel that they should be simple.

Still, a stunning 71% of startups use QuickBooks according to this BestVendor infographic. That is a higher market share than every other service including stalwarts like Salesforce, Google Apps, Google Analytics, etc. That 71% is compared to 5% who use Excel and 4% who use Xero.

Why?

Every third party financial service integrates with Quickbooks. One of the most time consuming aspects of accounting is getting data into the system so integration with every other service you use is critical. Even new services like Xero try to integrate with files exported specifically for QuickBooks (and do a poor job at it).

Not to mention that many of the competitors have lame limitations that severely impact their utility. For example, Freshbooks won’t email invoices (they’ll email links to invoices, but not the actual invoice).

We’re starting to reach the limits of what QuickBooks Online can do (deal with foreign currencies), and I’m definitely worried about the additional overhead we will inherit when we switch providers.

EarthClassMail for handling checks
If you are selling to enterprises, you will likely have to accept payments by check. Dealing with checks is another hassle that I’d prefer not to have to deal with. Among other annoyances, it requires the mailing and/or depositing of checks along with a paper trail to record the transaction history. Not to mention the hassle if they just get lost along the way.

EarthClassMail is a great solution to this problem. They provide a lockbox-like service that automates the receipt, scanning and deposit of checks. Just provide your customers a different billing address and checks hit your bank account faster than you could manually do yourself. The scans of checks and their related payment information also creates a perfect digital paper trail.

Nearly all banks provide lockbox services, but my experience with the ‘analysis checking’ services provided by banks has been universally miserable - they just have terrible UI. I can’t speak for all banks, so maybe there is a gem out there.

What solutions do you use? I’d love to improve on what we have.

My copy of Venture Deals arrived a little while back, but it wasn’t until my trip to SF this weekend that I had a chance to read it. My plan was to skim the book and then pass it on to one of my friends that is actively raising a round. Instead, I’m keeping it on my bookshelf next to the essential Entrepreneur’s Guide to Business Law and sending out a few copies.

I found myself alternately skimming sections and reading every single word. The book is succinct and doesn’t unnecessarily repeat itself. I was able to quickly determine whether a particular section had something new to teach me and dig in as needed. The book easily fit into a few uninterrupted hours on a flight, but I expect I’ll reference it from time to time.

As an early entrepreneur, a disproportionate share of your legal counsel’s time is effectively spent educating you. No sane entrepreneur should negotiate an obscure point on a term sheet that he doesn’t understand. In my experience, one of the reasons legal bills are often higher on a first-time entrepreneur’s company is this education curve. Read the book. The last thing you need is to shovel money from your completed financing to the lawyer that had to coach you through terms you could have quickly taught yourself.

But, the value of the book isn’t limited entirely to negotiating a venture round. I also found the tips on negotiation to be both timeless and more broadly applicable (I suppose there are negotiation books for that also). For example, most corporate legal processes are set up to exploit the tendencies of smaller companies. They take forever to process revisions and can involve many back-and-forth discussions. While I’m not particularly sensitive to the length of the process, I’m extremely sensitive to the time I spend on the process. Looking back, I see myself consistently agreeing to slightly worse terms the more time I spend on the negotiation. This wasn’t something I was aware of, and that self-reflection will be useful going forward.

Some of the biggest gaps in the book are easily addressed if you read it while online - I find that spreadsheets help me internalize the dynamics of financing terms much better than printed text, and there are plenty of resources for that online.

If you are operating (or hope to operate) a startup, Venture Deals is a great asset even if you don’t plan on raising any investments.

I received this awesome voicemail on my BrandVerity Google Voice number:

The Google Voice transcription wasn’t usable, so I did a quick transcription myself. You do really have to listen it to appreciate it though:

Hi this is Roberts, Data Service Provider for Google. I wanted to give you guys a call, we’ve got a pretty good opportunity in your marketplace. We’ve got some positions that opened up on the search engines Google, Yahoo and Bing and wanted to see if there’s any interest as far as improving your ranking on the search engines. When you get a couple of minutes give me a call. Phone number here is 800 219 6018…

I’ve fielded plenty of snakeoil SEO pitches before, but this one receives particularly high marks.

“It turns out the cloud is actually just some place in Virginia” — Tweet by jckhewitt

We were impacted by the long EBS outage at Amazon’s US-East-1 data center. BrandVerity uses a geographically distributed collection network to find and store ads and the sites they send traffic to for our customers. While less than a third of our servers are hosted by Amazon, we were significantly affected by the downtime because we use Amazon’s Elastic Block Store (EBS) to store some of our data. In total it was 28 hours before Amazon restored our EBS volumes. However, we were able to run a number of our systems during the outage by employing traditional disaster recovery techniques (restoring from backups, etc.).

We had a few expected and unexpected observations and lessons learned from the outage.

Unexpected Dependencies and Disappearing Redundancies
Many companies launched over the last few years run on AWS (over a third of Y-combinator startups have their principal domain at AWS Source). Supporting technologies that are especially useful to SAAS companies are disproportionately run on AWS.

We found unexpected dependencies in our technology vendors that we assumed would be independent of our own issues. An outside monitoring vendor had an AWS dependency that took down their monitoring. Not only was our own in-house monitoring impacted, but our backup monitoring went down completely.

Of course, the bigger issue is that Amazon’s Availability Zones weren’t as independent as Amazon has advertised. There is much discussion of this on the web at the moment and Amazon’s post-mortem provides an excellent view into the issue.

The outage has certainly raised the visibility of true multi-datacenter redundancy, which had been replaced in many organizations by multi-availability zone redundancy.

Cloud-based Outages Allow Teams to Focus on Software-based Recovery Options
Unplanned outages happen. In our prior companies, hardware outages have resulted in scrambles at the data center. Since the hardware issues would be the most immediate, nearly all technical personnel were on hand helping the systems administrators.

In this outage, we were a little uncomfortable to be a step removed from the core issues, but it allowed us to get back up and running more quickly.

Our team was able to focus on software-based recovery options. Rather than racking and unracking hardware, installing disks and operating systems, we were able to instead focus on multiple recovery options. We pursued two independent paths for recovery and were in a great position should our primary recovery option have run into unexpected problems.

Using Amazon AWS is (was?) Today’s Equivalent of “buying IBM”
When we selected Amazon’s cloud platform several years ago it hadn’t become the standard bearer it is today.
We were pleasantly surprised by our customer conversations the morning of the outage. Customers readily understood the source of the problems and very few of them attributed the issues directly to us. While we are fully responsible to our customers, they generally afforded us a wider margin and assigned more blame to Amazon.

I’ve had to do a few customer calls in the past where there wasn’t a third party involved in an outage. Even when the issue was an unlikely and unfortunate collision of events, they seemed to be generally unhappier with us.

Customers are certainly more likely to understand that Amazon was having a massive outage than the low probability nature of a near simultaneous failure of two disks in a Raid 5 volume.

We didn’t expect this response, but certainly didn’t mind it. It will be interesting to see if this perception will hold given this latest outage.

What have your lessons and observations been from the outage?

In tech circles the belief that consumer ISP DNS servers are terrible is widely accepted. So, I found this study fascinating. The quick takeaway is that Content Distribution Networks (CDNs) like Akamai rely heavily on the location of the DNS server to select the server to serve content from. Websites that rely on CDNs to serve some content load slower because the CDN chooses a suboptimal server. And nearly every major website uses CDNs for some of their content.

I ran the test script from my home comcast cable connection. In my case, the CDN server returned by OpenDNS was the slowest in both tests, while Google’s was the fastest. Here is the data from my test of Akamai’s CDN (summarized for readability):

>>> test(‘profile.ak.fbcdn.net’, ‘Akamai’) == testing : profile.ak.fbcdn.net Akamai =======
testing : OpenDNS ( 208.67.222.222 )
204.2.171.82

Ping statistics for 204.2.171.82:
Approximate round trip times in milli-seconds:
Minimum = 46ms, Maximum = 48ms, Average = 47ms
—————————————————
testing : Google ( 8.8.8.8 )
96.17.69.66

Ping statistics for 96.17.69.66:
Approximate round trip times in milli-seconds:
Minimum = 11ms, Maximum = 13ms, Average = 12ms
—————————————————
testing : default ( 68.87.69.150 )
96.17.70.16

Ping statistics for 96.17.70.16:
Approximate round trip times in milli-seconds:
Minimum = 36ms, Maximum = 38ms, Average = 37ms
—————————————————
=================================

On closer inspection, the Akamai node that serves my content varies by DNS solution used (I’m in Seattle):

DNS Provider	Akamai Node Used
OpenDNS	Los Angeles
Google	Seattle
Comast	Seattle

Conclusion

The author of the original post makes a great point. The location that CDNs use to serve content impacts browsing experience much more so than the speed at which DNS resolves an unknown domain. The DNS server used for the request can have a big impact in the selection made.

In my specific instance, Google’s DNS sends me to the best CDN node (at least for Akamai and Internap)

An IP Geolocation Aside

At Quova, we primarily sold against Akamai’s EdgeSuite service which provided some geolocation data. One of the common myths that Akamai sales staff liked to propagate was that Akamai’s data must be better because of the thousands of Akamai CDN servers.

We would usually respond that CDNs primarily depended on network topology, not user geography. Therefore the many CDN servers Akamai used provided no benefit to the core IP geolocation data. I think if I had known how much Akamai depends on DNS sever location, our counter-arguments might have been stronger.

I’ve been using a few services recently that I’m pretty excited about and wanted to share. I wanted to give a bit stronger of a testimonial than I could in a tweet so I’ve been collecting them into a blog post:

Interesting

• The biggest cloud provider is a botnet. Google manages ~1M CPUs, while the Connficker worm controls ~18M CPUs. Holy crap - this gives me an entirely new perspective on botnets.
• Snake Oil Scale. A great graphical look at various supplements and the extent to which science backs up their efficacy.

Cool

• TechStars Seattle Launches! It is great to see the Seattle entrepreneurial community come together to launch TechStars. The team behind this is fantastic. It will become a great launchpad for Seattle startups. Also, applications are now open.
• Seattle 2.0 New Management. Marcelo has done a fantastic job building Seattle 2.0 into the nexus of all things startup in Seattle, and I’m excited to see what the future holds for the brand!

Useful

• Rapportive: The best gmail firefox extension ever. Pulls your social media info for your email contacts into the sidebar (replaces the ads). It keeps creating serendipitous information discovery.
• HgInit: Stunningly written documentation on Mercurial by Joel Spolsky. I found the transition from svn to mercurial similarly difficult to Joel’s. This guide helped immensely. Many open source software suffers from terrible documentation. I wonder what other open source movements could achieve with similar efforts.
• Athlinks: Automatically gathers all of your race results (ok, most of them). On sign up, you are presented with a series of events and you check the ones that are yours. I’ve actively wanted this built for a long time. Not perfect yet, but I love the vision. A great showcase for what ‘intelligent agents’ (and likely a little bit of manual config) can do.

Every year my wife sends photo cards to our family and friends for the holidays. She carefully sorts through our photos from the year and selects several that she shares on the cards. She shuns the cards on photographic paper and instead selects heavy stock folded cards. The final product is usually great, however we’ve definitely been disappointed by print quality from time to time.

This year she used Tiny Prints (after a great experience with their birth announcements). Everything about the card was just better than the ones we had ordered in the past. But, the single biggest, unexpected awesomeness was the photo touch up they did on the cover photo of the card.

See the before and after below (yes, one version is scanned from the card). Sophie had a slight, but noticeable pimple to the right of her mouth in the original photo. It wasn’t something that we had thought to touch up, but we absolutely noticed that it had been removed when we got the actual card.

The amazing thing is that this is what Tiny Prints does. See co-founder Ed Han’s description from their ‘Our Story’ page:

Since our first year together, we’ve reviewed every order, edited every photo and obsessed over every typo and etiquette mistake on every card that has passed through our hands. After years of extreme perfectionism, our customers expect this level of attentive detail, and every member of our staff is proud to provide it.

I don’t think I could have been convinced in 2003 that there was a market for another online photo card company. Ofoto, Shutterfly and Snapfish seemed to have that market pretty locked up. However, there was clearly a high end to the market that had been overlooked and Tiny Prints has grown aggressively since. They initially launched with a focus on birth announcements, and have steadily expanded to a whole range of cards, calendars, photo books and even corporate cards (no save the date magnets though).

At this point, I can’t imagine we’d order cards from anywhere else. There aren’t too many businesses that fall into that category.