Satya's blog

I worked at a company that was spending over $70k/month in AWS EC2. I brought that down to $25k/month. Here's how.

The company had been building hardened EC2 virtual machine images (AMIs) for several years. They built images for several variants of operating systems. They built each set every day. But, once a month, the latest set was promoted to "gold" status.

The other, non-promoted, images had been sitting around forever. There was no cleanup, no archival.

Notice that the daily builds were mostly useless. At best they served to show that the pipeline was still working. An expensive canary, as we were cranking up some rather large instances to build the bigger variants.

The daily builds were also used for testing as developers built new images... but developers _also_ kicked off image builds manually, so this wasn't necessary.

I did two things: I built an image sweeper to delete old, non-gold images, and I reduced the frequency of image builds.

The image sweeper was built conservatively, so it wouldn't delete unless it was sure the image wasn't "gold". I had to check several places, because of course the location of "gold" images listings had changed over the years. This sweeper became an on-going cron job (triggered by EventBridge). Additionally, I enabled the EC2 recycle bin so that my first runs of the sweeper had a safety net.

When I reduced build frequency, at first I made it something like once a week (plus end of the month). The "end of the month" date differs, of course, so I made the builds happen something like the 25th, 27th, 28th, 30th. Eventually -- months later -- the requirements solidified into "once a month" so we could reduce the build frequency down to the minimum.

Old advice was to update your resume and cover letter for each application. Ain't nobody got time for that! Because all that work leads to a low-effort, generic, "we're going with other candidates" response.

Before you get anywhere you have to log in. To Greenhouse or Workday or ICIMS. Again. With a new account. With a strong password. Yes I have a password manager. It is littered with such logins. See previous point about low-effort rejections.

Now every application needs a resume, cover letter, at least 4* questions about gender, ethnicity, disability, veteran status, and visa status... sometimes repeated during the _same_ application!

Why do they need my home address at this point?

Then you get questions about your salary expectation. One place I tried to apply, the response field was a drop down with the answers yes and no. Uh. And -- this might be LinkedIn's problem -- the university drop-down with _every_ school around the world (literally): _not_ alphabetically sorted.

Then you get "why do you want to work here" type questions. Because I want health insurance, I guess. Oh no, wait, that's not good enough, you have to be passionate about whatever they do!

* The 2 hard problems in programming are: cache invalidation, naming, and off-by-one errors.

In day-to-day programming, I rarely think about algorithms and data structures. Binary trees. Sorting.

That's because algorithms are already encapsulated in the standard libraries of the programming languages I use. Most languages like Python, Ruby, and Golang have sort functions.

For most data structure purposes, lists and associative arrays (dictionaries, aka key-value) are enough.

Faster algorithms? Processors are relatively cheap, parallelize it.

Memory? Memory is cheap and if that's not enough, spool it out to disk. If that's slow, parallelize it.

No, what's expensive today is network calls. And that's only because everything's fast enough that we can make API calls -- that's Application Programming Interface -- over the Internet.

Of course there are still situations where algorithms and data structures matter. The preceding just talks about 90% of general programming these days: websites and backends. Someone still has to make the standard and not-so-standard libraries of all these languages work, and if you're in embedded-devices land then all bets are off.

For those that don't know, and those that do, git bisect is a very cool tool/strategy/usage to find out the answer to "which commit broke everything?" I used it recently. I have a page which suddenly started showing blanks instead of the numbers that should be there. So I used git bisect. First I did `git bisect start`.

I knew this was on my "test" branch, so I checked that master was good (it was), and that the tip of test was bad (it was), so I marked both of them with `git bisect good` and `git bisect bad`.

Then I just kept reloading the page, marking each commit as good or bad depending on whether the numbers showed or not.

`git bisect` does the work of checking out the "next" commit after you name two commits good and bad. It does a binary search, so it checks out a commit half-way between the last-marked good and bad ones, and if you mark that one good/bad it moves HEAD to half-way between that one and bad/good. It's cool to watch in action.

https://git-scm.com/docs/git-bisect

You can also use this to find the "commit that introduced the change", it doesn't have to be a breaking change. Just so long as you have a quick way to test whether a particular commit falls on the "good" or "bad" side. In my case, that was just reloading the page to see if numbers showed up. Other ways are to run a unit test, run a script, or whatever.

I want a Smart car!

http://www.zapworld.com/cars/smartCar.asp (dead link, new https://boxercycles.com/zap-electric-vehicles/ )

I wish it was cheaper, though :-(

There are many things that people call "Agile software development". Here's how I do it. It's similar to how Pivotal Labs did it (and for all I know, still do).

Have a force-ranked issue system

Issues (tickets, bugs, stories, whatever you call them) should be in a single list. The list should be ordered by priority in which your project manager wants things done. That way the people working on things can always just pull the top-most item off the list. Yes, please self-assign items as you start work on them so others know that it is being worked on.

There should be one list per team or whatever unit of organization you have. Any member of the team can pick up the next item. There is no confusion about which team owns an item.

Please include tech-debt items! And please allow ICs to add items.

What counts as a ticket/issue story?

Preferably include smaller self-contained features or bugs. Try not to put different things in the same issue, even if related or "in the same area". This is of course subjective, but one rule-of-thumb is "can it be tested by the PM or QA independently?"

Sprint

Sprint length of two weeks seems ideal. One week is too short, four is too long. Three is just odd.

Sprint lifecycle should be something like: backlog grooming before the sprint, planning meeting on the first day (or before), check-ins/standups during, and a retro on the last day.

Backlog grooming: PM, EM, tech leads, and maybe some senior individual contributors (depends on size of team) can go through the list of issues, maybe triage if needed, re-order them if needed. Basically, prepare for the actual planning meeting. Ideally backlog grooming is ongoing, but sometimes it doesn't happen. Should be a 30-minute or less meeting.

Please ensure a healthy, non-zero, number of tech-debt/IC-contributed items are addressed. This is a good time to ask for more info from stakeholders or to close stale issues.

Planning: Whole team, issues should be quickly discussed so the whole team has a general idea of what's involved. Issues can be assigned now, but try to let ICs self-assign rather than handing them down. Should be an hour or less.

Check-ins/standups: Daily fast standup or every-other-day check-ins to ask and answer "are we on target?" (and adjust stakeholder expectations).

If it's a daily fast standup, each person should quickly (within 30 seconds -- do NOT time people) tell what they got done yesterday, what they intend to work on today, and importantly, any blockers or requests for help. No interruptions, jokes, side-tracking, or requests for clarification. Break out after the stand-up if any of this is desired, and no one has any obligation to stay for this followup. Note that this is often done asynchronously in the team chat channel.

The check-ins or standups shouldn't be more than 15 minutes; if you're always going over, you have a big team or a talkative team. Try to find a time that works for everyone. Just before lunch is good motivation!

Retro: Various teams use various forms of retro, but the core seems to be "what went well" and "what we could do better". Let everyone contribute by writing their lists all together on a board or paper or something, and then going around one by one to expand verbally. Shouldn't take more than 2 minutes each, adjust for team size. Try to get action items for improvements. Try to always have some positive items.

The retro can segue into backlog grooming or even planning for the next sprint.

Suppose you use an issue tracker like JIRA or Pivotal Tracker (PT). Here's how I use it effectively.

Stories, short for "user stories", issues, and tickets are used interchangeably here. There are subtle differences (or not so subtle ones) that don't matter for this discussion.

In JIRA you have multiple projects. Each project has a list of tickets. This should be called the "Backlog". Additionally, you can create a "Sprint".

In PT, in the default view, there are three columns: the "current sprint", the "backlog", and an "icebox". I'd advise finding the setting to enable stacking of current on top of backlog, because that gives a "truer" view.

The (current) sprint contains stories that are being worked on in this sprint. These stories will be in various states, such as Done, In Review, In Progress, To Do. Those are the 4 major states for stories. PT enforces the states while JIRA is more configurable. PT places stories in the sprint automatically based on project velocity (which is based on story points). In JIRA, the current sprint is manually determined during your sprint planning meeting (aka iteration planning (IPM)).

Every story in the current sprint must have a story owner and (if your team uses estimates) an estimate. Some shops do not assign a story owner until someone starts the story, and then that person is the owner. This works well in smaller teams consisting of mostly senior devs. PT enables this by auto-assigning owners.

Additionally, stories may belong to epics. That's a good way in JIRA to group stories. PT also has epics but I prefer to use Labels in PT. JIRA has labels but they're less visible to me. I'd advise using Components in JIRA for the same purpose.

The Backlog contains stories for upcoming sprints. They may be un-assigned, un-estimated, not even fully fleshed-out. All of those things can happen during the sprint planning meeting.

PT has an additional space, the Icebox. This is sort-of the ideas bin: stories that haven't been thought through, or that are a reminder to actually add a "real" story, or stuff that's deferred until some later date.

If team members run out of stories for the current sprint, they may be able to pick up stories from the backlog. After checking with the project manager, perhaps. Beware that stories in the backlog may not be quite ready to be worked on.

PT forces all the stories to be in force-ranked order, which means someone (project manager, in consultation with senior developers) needs to order the stories. JIRA can also be configured to do force-ranking. Stories should usually be placed in order of business priority and according to depenecies. Feature Y has priority but also depends on Feature X? Place X first (higher) in the sprint.

(Why the PM? That's who's running the project. Why the senior devs? Because that's who can point out hidden dependencies. In a small enough team, it'd be all the devs... which means you're having a sprint planning meeting. Or a backlog-grooming meeting, which is a pre-planning meeting. Because of time constraints. These meetings need to be extremely focussed, and I will write a separate article about them.)

Here's a simple git branching workflow that I like and use.

All "production" work is on the master branch.

Start a feature, first create a new branch:

git checkout -b feature-name

Perhaps prefixed or suffixed with a ticket/issue identifier, e.g. axp-1090-add-phone-field

Usually useful to push the branch to your git origin (often, GitHub) right away, and have git track the remote. This is done in one command with:

git push -u origin HEAD

HEAD is a special thingy referring to the commit we're "on" now.

Do the work. Maybe multiple commits, maybe some work-in-progress commits.

git add
git add -p
git commit -m "AXP-1090 Add phone field"

add with -p makes git prompt for each change. Useful to review the changes and to be selective. Maybe some changes should be in a different commit?

The git commit command can open a text editor for the commit message, or you can specify the message as shown here.

To re-order commits, use `git rebase -i` which will rebase against the upstream, in this case master.

Move the commits as presented in the editor in the order you want. Note that work-in-progress commits can be squashed by changing "pick" to "squash". Each squashed commit will become part of the commit above it. And yes you can squash multiple commits in a row.

Solve any conflicts. In the default configuration git should show you lots of help on how.

To integrate any changes that have happened in master, every now and then (with a clean branch! see `git status`) do a fetch and rebase (and do one of these when you're ready to merge your changes back to master):

master:  M1 -> M2 -> M3
              \
your branch:    B1 -> B2

git fetch
git rebase

master:  M1 -> M2 -> M3
                      \
your branch:           B1 -> B2

Beware that if you switch to master (git checkout master) you will still be behind origin/master, and need to do `git rebase` *on master*.

When the branch is in a sufficiently clean state, push your work to the remote:

git push

If you've rebased, you'll need to use `git push -f` i.e. "force-push". And that is why we use branches, and that is why everyone's should be on their own branch. Otherwise, force-pushing will overwrite other people's work on the branch. That is why we never force-push master (except when we do).

Use `git status` often, which is why I have a shell alias `gs` for `git status`. And I have `git` itself aliased as `g`, with various git commands shortened in my ~/.gitconfig

To merge your changes to master, open a Pull Request on GitHub or, if you're doing it yourself manually, you can merge. First, rebase against master, then switch to master and then merge your branch.

git rebase
git checkout master
git merge axp-1090-add-phone-field
git branch -d axp-1090-add-phone-field # optional, delete your branch

At this point you should have a clean master that you can push (not force-push) to origin.

• Learn git branching: https://learngitbranching.js.org/?demo
• How to Write a Git Commit Message: https://chris.beams.io/posts/git-commit/

Databricks is a web-based Spark notebook system, and also the company that wrote several spark libraries. The notebooks support Python and Scala.

Databricks runs your jobs in your AWS account, on EC2 instances that Databricks manages. The initial setup is a little tricky.

Once everything it set up, it's a pretty good system for developing and debugging Spark-based jobs. It can run jobs on a cron-like schedule. Databricks has support for retrying failed jobs, and can notify about success/failure by email. It gives good access to the spark worker logs for debugging.

However, this is expensive to run. I came up with a workflow that involves development/debugging on databricks, and then export the notebook as a script to be run in EMR (Elastic Map-Reduce, an AWS product). I use Python and pyspark, so this works pretty well.

I do need to import the libraries that Databricks imports automatically. I forget the exact import statements I used, but they are easy enough to figure out.

I used Luigi, which is a Python library for setting up task workflows (see also: Airflow). I set up a few abstract classes to encapsulate "this is an EMR job". Then I extended one of those classes to actually run a spark job, something like:

class ActualJob(AbstractSparkJob):
    luigi_param = luigi.Parameter()  # I usually pass a timestamp to every task

    def spark_cmd:
        return "this_command.py --runs --spark-script --located-in s3://example.com/foo.py"

spark_cmd returns a command line string.

My AbstractSparkJob takes the given command and does one of two things: either submit an EMR step to the EMR cluster, using command_runner.jar to run the command, or ssh into the cluster and run spark-submit with the given command as its parameters.

(I don't have all the code do that available to post right now, sorry)

The abstract classes encapsulate all the work of spinning up an EMR cluster (and shutting it down), and making the AWS API calls via the boto library, and the work to ssh in and run spark-submit.

The abstract classes makes it easy for any other data infrastructure developer person to add more jobs.

The actual script that was exported from Databricks lives in S3 and is referenced by the command shown above. EMR does the work of fetching it from S3 and running it. Part of my data pipeline startup is to copy all the EMR scripts, which I keep in a convenient subdirectory, up to a specific prefix in S3. That way my actual Spark scripts all live in the same code repository as the rest of the pipeline.

Component	Wattage
CPU+RAM+fans	23W 210mA
+ HDD	37W 240mA
+ GTX 1050	57W 500mA

Connect the phone via USB. Go into Settings, About Phone, click on Build Version about 7 times. Then Developer Options becomes available, use that to turn on USB Debugging.

Use `lsusb` to get the phone's code. Example output:

Bus 001 Device 059: ID 04e8:6866 Samsung Electronics Co., Ltd GT-I9...

Create a file or directory `~/.android/adbusb.ini` containing the line "0x04e8" based on the lsusb output above.

As root, drop a file called /etc/udev/rules.d/s4-android.rules containing the following:

SUBSYSTEM=="usb", SYSFS{idVendor}=="04e8", MODE="0666"

Note that the vendor id is again from the lsusb above.

Run this too (as root):

chmod 644 /etc/udev/rules.d/s4-android.rules

Chef recipe for that file:

file 'udev_rule_s4_android' do
    path '/etc/udev/rules.d/s4-android.rules'
    mode '0644'
    owner 'root'
    group 'root'
end

execute 'udev-restart' do
    command '/etc/init.d/udev restart'
    subscribes :create, 'udev_rule_s4_android', :immediately
    action :nothing
end

Connect the phone in 'PTP' mode:
Connect the phone via USB. Drag down the notifications, click on the 'Connected as...' notification, and set it to PTP or Camera mode.

`adb shell` should now get you a shell on the phone.

References:

• http://stackoverflow.com/questions/5510284/adb-devices-command-not-working

• https://ubuntuforums.org/showthread.php?t=2298370
• http://askubuntu.com/questions/213874/how-to-configure-adb-access-for-android-devices
• http://android.stackexchange.com/questions/21112/new-phone-how-to-transfer-game-progress