Guest-Lecture at Johns Hopkins

Air Force,Analysis,Geolocation by on February 13, 2008 at 11:44 am

I guest-lectured at Johns Hopkins last Saturday in the Analysis, Data Mining & Discovery Informatics Class in the Intelligence Analysis Master’s Degree program. The students were largely early-career intelligence analysts that are accustomed to conducting one-off analysis, but rarely think about building analytic systems.

I gave an overview of IP geolocation, had the students test their skills at mapping IP addresses (thanks for the IPs Bob!), and then we discussed how we might build an analytic system to map IP addresses.

My takeaway from the class was that the government doesn’t train its people how to build systems. The analysts that I worked with had all been in their roles for a number of years, but this course seemed to be the first time that most of them had ever thought about improving the analytic system, not just improving their personal analytic capabilities

I’ve found the culture of the intelligence departments to be very analyst driven. The analyst is at the top of the food chain. Tools are built and data is collected to improve the analyst’s job.

However, analysts frequently do things that can be automated. Even worse, the agencies reinforce this by building tools that make their automatable work easier to do manually.

The key is to realize that the analysts are components of a larger system, and not the purpose of the system. We found a great balance at Quova where our analysts and algorithms worked closely together, creating a system much stronger than any one component.

Suvival of the UnFittest

Air Force by on May 12, 2007 at 7:23 am

I’m out in DC this weekend, so this post touches on another government-related topic.

The government’s budgeting system creates completely perverse incentives. Budgeting is done annually in a trickle down (up) fashion. Congress decides that the Air Force gets ~$100B. This pot then gets divided up by the Air Force to component commands, that then allocate to wings, which allocate to groups, then squadrons, then flights. Upwards requests for funds are also made and aggregated and these requests travel up the tree.

The brain-dead, low risk way to allocate funds down the tree is to take last year’s budget and add a little bit. No one likes to have their budget cut, and no one likes to see budget requests skyrocket. So, the scenario below is played out over and over:

Tim runs the Services Squadron. He’s shrewdly managed his budget by making several great decisions. He ended a major contract with an expensive food supplier, and replaced it with a supplier that could provide lean, just-in-time delivery. He introduced pre-made sandwiches at lunch. This reduced the labor costs at the made-to-order sandwich station and increased the capacity of the lunch room. Two months before the end of the year, Tim was on track to be 10% below budget.

Major Spender runs the Communication Squadron. He’s made a number bad decisions. He brought in contractors to build an expensive content firewall to block all webmail, sports, games, social networks, etc. Instantly people began to complain. The firewall was blocking access to weather information, wikipedia and to news sites (that had sports scores). Rather than admit a mistake, Major Spender created a ‘site inclusion’ process. People that wanted access to a website, would fill out a form justifying their need for access to the site. The flood of requests overwhelmed his small staff and degraded service, but he was still on target to spend all his budget.

The General that runs the base learns his base is 5% under budget. He knows that the budgeting process for next year starts with this year’s spend, so he has to spend the rest of this year’s budget. He calls Major Spender into his office and asks him how his budget looks. Major Spender tells him that he needs a second firewall and he needs contractors to manage all the firewall inclusion requests. The General gives the Major authority to spend an additional $500K as long as he can spend it by the end of the year (which he promptly does).

A month later, the General is planning next year’s budget. Tim presents a budget 20% higher than last year’s. He’d like to upgrade his purchasing software so that he can track the spoilage rates of his perishable foods, plan for soldier surges and predictably purchase supplies. Although this will be an expensive purchase he expects it will save the wing money in subsequent years.

Major Spender presents a budget that is identical to the amount he spent last year (which was higher than his budget). He justifies it by claiming that was the fixed cost to run the base’s communications.

The General approves Major Spenders budget, but not Tim’s. Tim’s budget would require the general to ask for more money than he spent last year, and the General knows that is frowned upon. Tim has his budget reduced to what he spent last year (which is actually a lower budget than he had last year).

Major Spender uses the new budget to upgrade the computers of all of the senior officers on the base. The senior officers are happy and the Major’s name gets mentioned favorably in office circles. When the promotion cycle comes around in 6 months, Major Spender gets promoted because he ‘gets things done’. Tim isn’t promoted and grows increasingly frustrated.

This cycle happens over and over again at every level. Budgets get spent because they will be lost if they aren’t spent. Officers that spend their budget and then quickly spend ‘overage’ dollars get promoted. Eventually the efficient managers are driven out of the system and the inefficient managers get larger and larger budgets.

This continues unabated until one thing happens: Budgets get reduced. The capabilities that have been bred out of the system suddenly become critical.

This is unfolding within the organization I support in the Intelligence Community. A brilliant senior manager is using slight budget constraints to literally squeeze fat out of the system. Government employees are howling. Contractors are howling. Everyone is howling. But, slowly change is happening. Inefficient projects are dropped. Proposals are reconsidered. Systems are shut down. I literally saw a $50,000 contractor proposal to convert XML search results to an RSS feed. Nonsense like this is being tossed where it belongs.

The Intelligence Community’s Retread Problem

Air Force by on April 20, 2007 at 7:25 pm

I spent close to a week out in DC with my Reserve Unit (sorry for the blogging - hiatus). I continue to be amazed by the inefficient nature of government spending - particularly in software and related services. My exposure has primarily been to the systems within the Intelligence Community (IC), however I’m sure that other agencies suffer from similar problems.

The process begins in an awful place. The gov’t becomes convinced that their need requires custom requirements… because, no other organization in the world is like the US government. Government contractors reinforce this belief and set off building a custom application on the most expensive software and hardware available. In several years, the government gets a multi-million dollar system that could have been replaced by a $30K system in a month that meets 95% of the business needs.

On top of this, the IC suffers from a severe lack of people that know better. It isn’t that IC personnel aren’t smart (many are brilliant), it is just they’ve only known the ‘government way’ their entire careers. The IC personnel come from one of three channels:

  • Government employees
  • Military members
  • Contractors

Government employees and military members are typically lifers. It is very rare that you see someone mid-career move into these positions. It can take up to two years to get a security clearance. This barrier to entry prevents people from joining mid-career. Government employees typically begin their careers in the government (often getting their clearance while in school).

But what about the contractors? They’re outside the system right? Well, no. It’s that pesky security clearance again. Most of the big contracting firms hire from within the system, pulling in govies and ex-mils that are already cleared. They then sell them back to the government at far higher rates. Hence the term ‘retread’. The system feeds on itself, rarely introducing fresh thoughts into the system.

What’s the solution?

Clearly barriers need to be lowered to get new people into the system. A few approaches:

  • Carve out unclassified work. Unnecessarily classified projects significantly limits the talent pool that can contribute. There are plenty of projects or components of projects that don’t need to be classified.
  • Overhaul acquisition laws. Navigating the federal acquisition process is extraordinarily challenging. Companies that contract to the government often have dedicated personnel to navigate the process. The goal is to get “new to the government” people helping think about our projects. I’ve commented before on the backwards nature of the FAR.
  • Expedite clearances. I know that background investigations are important and take time, but we simply have to be able to clear people faster. Two years is too long…

The machine cares

Air Force by on March 8, 2007 at 9:49 am

Sumit forwarded me this note from the Air Force.

From: System Generated Email
Date: Mar 8, 2007 5:01 PM
Subject: Record Review Notice

*** Do Not Reply —- System generated email —- Do Not Reply ***

Dear Service Member,

On the occasion of your birthday, you are invited to review your automated personnel record at the Virtual Military Personnel Flight (vMPF) website: Please check over your record information carefully. The reflected information is current data contained within your Military Personnel Data System (MilPDS) record and affects such things as assignments, promotions,
classification actions and other career related items. If you discover problems, print a hardcopy using the printer friendly icon, annotate the problem areas and then contact your local Commander Support Staff or Military Personnel Flight to get the errors corrected as soon as possible.

Air National Guard and Air Force Reserve (to include AGR/Stat Tour) members: By law, you are required to validate your Civilian Employer data at the following OSD website: ttps://

The best part is that this is that the message has been the same for several years.

Happy Birthday Sumit.

What is cybersecurity’s PR problem?

Air Force,Security by on February 17, 2007 at 7:00 pm

Slashdot picked up an FCW article on coordinated Chinese hacking.

The full article is worth a read, but here are a few standout quotes:

Attacks coming from China, probably with government support, far outstrip other attackers in terms of volume, proficiency and sophistication, said a senior Netwarcom official, who spoke to reporters on background Feb 12. The conflict has reached the level of a campaign-style, force-on-force engagement, he said.


Current U.S. cyber warfare strategy is dysfunctional, said Gen. James Cartwright, commander of the Strategic Command (Stratcom), in a speech at the Air Warfare Symposium in Orlando, Fla., last week.

Although the level of discourse at Slashdot is not always the highest, I was disappointed that this audience did not appreciate fully appreciate the problem. Many indicated that they didn’t believe there was a problem.

So, if Slashdot readers can’t grasp the threat how can we expect politicians to get it?

The weaknesses of our cybersecurity systems are well established - just about every red vs blue penetration test the US runs finds countless flaws. Military leaders are beginning to understand the risks (as evidenced in the article above). However, law, policy and systems continue to lag well behind the capabilities and risks of technology.

Sumit and I contributed to a ‘National Cybersecurity Strategy’ Paper about a year ago that was meant for the White House. We worked on developing the business case for why cybersecurity matters and what the risks look like. The reality is that we haven’t seen a large-scale coordinated attack intended to cripple. The types of threats we’ve seen to date have been much more akin to ‘tests’, ‘training activities’ and intelligence gathering (and all very successful).

There are several problems that the cybersecurity practioners suffer from:

  • All of the practioners hide behind email addresses only accessbile on secure netorks and phones with no answering machines. They also aren’t known for being the most social people.
  • All “events” are classified and even those leaked to the public end up are said to be bastardized versions of real events: The Invasion of the Chinese Cyberspies ( You can’t convince someone of a problem if you can’t tell them about it.
  • We’ve never had a truly crippling attack. The capabilities exist and the vulnerabilities are there. Government policies are notoriously reactive…
  • Bastardized priorities. Deep down the chain of command, DOD system admins came to believe it was their mission to filter websites and otherwise hinder the use of the ‘Internets’. This deeply misguided prioritization has alienated most of the rest of the government (who aren’t able to get to their yahoo mail accounts or check their sports scores).

I don’t know what it will take to bring about serious change. It will have to happen at some point - I just hope it is proactive change.

A Reservist, 30 days & $45K – How a reservist outprocured a $15M Defense Program

Air Force by on February 12, 2007 at 12:23 am

The military has some of the most inefficient procurement systems I’ve ever seen. I’ve commented before on how federal acquisition laws primarily benefit contractors, but the inefficiency of these systems is staggering.

There was a great article this week in the WSJ (this one appears to be free) about:

a Marine officer in Iraq, a small network-design company in California, a nonprofit troop-support group, a blogger and other undeterrable folk designed a handheld insurgent-identification device, built it, shipped it and deployed it in Anbar province. They did this in 30 days…

30 days. Yeah, there were several herculean efforts that made this possible, but um, this is a war.

There is an existing government program to do exactly this. It was established in 2004 and funded $5M in its initial year, and who knows how much in subsequent years. As far as I can tell, the system is far from operational, and far from being useful in Iraq.

Maj West, the reservist mentioned in the WSJ article, is an energy trader for Goldman Sachs in his real career. Most government lifers (military, civilian and contractors), don’t think the same way that the private sector does. Designing, building AND delivering a biometric device like this in 30 days is outside anything the lifers could ever conceive. I’m fairly confident that this would never have been done by a traditional military officer.

Most active duty generals and civilian leaders look at the reserve forces as simple manpower. However, the backgrounds of the reservists are far more diverse than anything else you find in the military and they bring far more creative solutions to problems than is taught to career officers.

Absurd Password Requirements

Air Force,Security by on January 28, 2007 at 11:09 am

The Air Force runs a portal that all its members can access. It actually aggregates a bunch of really useful information such as pay history (limited to 2-3 months), white pages of personnel, and various personal personnel data and functions. I access it once every 2 months or so.

They make you change your password once every 90 days. Here is their current password policy:

Passwords Must:

  • Be a minimum of nine characters in length
  • Contain each of the following in the first nine characters:
    • Two Uppercase Letters
    • Two Lowercase Letters
    • Two Special Characters (except ? which is reserved)
    • Two Numeric Characters

Seriously. I’m all in favor of strong passwords, but this is absurd. It would seem like passwords like this are more secure. There are two flaws to this logic:

  • Users write them down. No one is going to commit this obscene password combination to memory. They’ll write it down, email to themselves or save it on a file. I’d be willing to bet that you can find stickies with this password on the monitor of a substantial number of personnel. My login procedure now includes all of the reset password steps because I can’t ever remember my password.
  • Users will use patterns to remember the password. Once a password gets this complex, users resort to finding patterns on their keyboard. I’m sure 12#$QWert is commonly used. The password just became incredibly easier to solve via brute force.

If your security needs demand this complex of a password, don’t let users choose them. Assign them a password that is randomly generated. You’ll at least reduce the second vulnerability (the more dangerous one).

What does the NSA do?

Air Force by on January 10, 2007 at 9:10 pm

There are plenty of misconceptions about the NSA, chief among them is what the NSA does. I love the various Hollywood misconceptions (Sneakers Enemy of the State, Mercury Rising), as well as the many shocked blog posts every time the NSA puts out a security guide, or, gasp, helps industry secure their shit.

The NSA is divided into two organizations:

  1. Signals intelligence. This is the role most of us think of w.r.t. the NSA. They are responsible for deriving intelligence from signals (phones, faxes, computers, etc) this is notably different form human intelligence (the CIA’s stomping ground). They are also responsible for code breaking, and are said to employ the most mathematicians of any organization in the US.
  2. Information assurance. Protecting US government systems, especially all classified and sensitive information. Intrusion monitoring and response, penetration testing, cryptography support (protecting us from other snoops). Imagine trying to protect the computers used by several million high school and college kids. Yeah, that’s about what they get to do.

Next time you see a security guide from the NSA or a report that they helped CompanyA secure their software understand that its in their best interest to actually secure systems.

Next Page »
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License. | Dave Naffziger's BlogDave & Iva Naffziger