Satya's blog

Feb 15 2016 22:03 Moving things in AWS S3

Recently I had several hundred small files in an AWS S3 bucket, in folders by date.

Something like s3://bucket/2016-02-08/ (and a couple layers deeper), with a few "directories" under the dated "directory". (The sub-directories were the first letter of the ... things, so I had about 62 sub-directories (A-Z, a-z, 0-9). This is a naive hash function.)

I wanted to move them into a year-based "directory", so s3://bucket/2016/2016-02-08/

(Why am I quoting the word "directory"? Because they're not really directories, they're "prefixes". This is relevant if you use the AWS S3 SDK libraries.)

Moving them via the S3 web "console"'s cut/paste interface is slow. REALLLLLY slow. Like, multiple-days slow.

So I (after trying a few other things) pulled out the aws command-line tool (AWS CLI).

Since the sub-directories were letters and numbers, I could do this:

`for x in a b c d;do echo aws s3 mv --recursive s3://bucket/2016-02-08/$x s3://bucket/2016/2016-02-08/$x \&;done >`

The `for` loop runs through a, b, c, d (different from A, B, C, D), and sets up a recursive move operation. This move is much faster using the AWS CLI. Additionally, I background the process of moving the 'a's (using the `\&`) so the 'b's can start right away, and so forth.

But I don't run the commands right away. Notice that they're being `echo`ed. Capture the output in a file, and run the Why?

Because I can now set up a second file with d e f g, to go right after the first, or even in parallel. So now I have up to 4 or 8 move operations going at once. watch the whole thing with `watch "ps axww|grep scr"` in a separate terminal, of course.

But mainly because the `&` backgrounding interacts weirdly with the for loop.

With this, I was done in well, a couple of hours. A lot of that was waiting for the last copy-paste I ran in the web console to finish.

Mar 05 2015 16:56 A front-end developer is me

I'm not really a front-end developer, more "full-stack" (HTML, CSS, JS, server-side, unix-command-line capable, database-y, etc.). I say, though, that my strengths lie in trouble-shooting and re-factoring.

So today it only took (less than) ten minutes to determine why the layout was broken, with slight familiarity with the HTML and none with the CSS in question. We had 4 divs in a row, and there were supposed to be four more in the next row (rows being defined in CSS; the HTML just had a bunch of div tags).

div div div div
div div div div
div div div div
div div div

The second row only had 1 div showing in it, at the far right, so it appeared like divs number 5, 6, 7, were missing. I caught on pretty quick that they were all floated divs, and so most likely the one div in the first row that was different (different CTA (call to action)) was the issue. Sure enough, the CTA in the 3rd div was 1 pixel taller than the rest.

Since I was using Chrome's developer tools to poke at it anyway, I set the CTA's height 1 pixel lower, and everything suddenly lined up. Passed that info to the front-end dev, who fixed it.

Nov 02 2014 10:07 Flash video encoding with mencoder

I use mencoder to encode Flash videos for various reasons. Here's a command I use for videos of the kids:

mencoder $1  \
-ofps 30 \
-o $1.flv  \
-of lavf  \
-oac mp3lame \
-lameopts abr:br=64 \
-srate 22050 \
-ovc lavc  \
-lavcopts vcodec=flv:keyint=50:vbitrate=600:mbd=2:mv0:trell:v4mv:cbp:last_pred=3 \
-vf scale=480:320

That gives 30 frames/sec with a video bitrate of 600 bits/sec, and audio sample rate of 22050. For a webinar, which has fairly static screen images, I drop the frame rate to 10 or so (-ofps 10), set a bigger video size (-vf scale=720:480), and change the keyframe interval to 250:

mencoder $1  \
-ofps 10 \
-o $1.flv  \
-of lavf  \
-oac mp3lame \
-lameopts abr:br=64 \
-srate 22050 \
-ovc lavc  \
-lavcopts vcodec=flv:keyint=250:vbitrate=600:mbd=2:mv0:trell:v4mv:cbp:last_pred=3 \
-vf scale=720:480  \

Other possible options:
-speed 1.05 (change the speed, 1=original speed)
-vf-add rotate=1 (rotate the video; experiment with the numbers or read the mencoder man page)

Tag: howto

Oct 18 2014 07:26 git branches and git add -p

So you made some changes and want to commit to two different branches?

You have this:

$ git status

modified: foo
modified: bar

You want foo on branch A and bar on branch B. Easy enough:

$ git co A
$ git add foo
$ git commit
$ git co B
$ git add bar
$ git commit

You want foo on branch A along with one of your changes from bar. You want the other changes in file bar on branch B.

$ git co A
$ git add foo
$ git add -p bar
$ git commit
$ git co B
$ git add bar
$ git commit

Note the "git add -p bar". At that point, git will ask you, interactively, which changes you want to include in the changeset.

What if branch B doesn't exist yet? Instead of simply doing 'git co B' you would do 'git co -b B'.

At some point, you have a local branch that has never been pushed to the origin (that's just the usual name for the primary remote repo, btw). You can push the branch like this:

$ git push origin B

That pushes the current branch to a branch named B on origin. You could be on branch B, and do this:

$ git push origin C

And you've just pushed to a branch named C, and confused your future self.

Suppose you want to set up the remote branch and the local branch should track it. So that git status will show you things like "Your branch is ahead of origin/B by 10 commits [ you should push]" and also so that you can, in future, simply run `git push`. Then you'd do this on the first push:

$ git push -u origin B

And subsequent pushes would just be:

$ git push

Tag: howto git

Jun 29 2014 22:40 Git branching tips

When creating a branch with git:

git co -b new_branch_name

This says make a branch named new_branch_name and switch to it.

When the time comes to push it to your remote server:

git push -u origin new_branch_name

This says push the current branch to origin (your default remote server) and call it new_branch_name on origin, and also set up to track that remote branch (-u). Tracking means, when you're on this branch and do git pull, git knows which branch to merge into this one. And git status tells you useful things like "Your branch is ahead of origin by 1 commit".

Tag: git

Mar 23 2014 09:33 Vim
"The composability of vim commands is a beautiful thing."

agumonkey on HN

Meaning that you can put vim commands together to make new commands.

Dec 08 2013 08:28 VIM encoding
So you just pasted from Excel into vim and now there's unicode in your ACSII? The easiest way to fix that is to type
set encoding=iso-8859-1
at the colon prompt. (Yes, I know I said ASCII.) You'll be able to see the extra unicode bytes and delete them.
Dec 16 2012 07:55 CLUI: Command Line User Interface
CLUI: Command line user interface. One of those comamnd-line based interfaces, usually built with ncurses, that provides a simplified GUI so you don't have to type actual commands. Example: aptitude, midnight commander, pine, mutt.

Tag: geeky

Nov 08 2012 22:08 Desktops
My desk, running Windows, Mac, and Linux all at the same time on 3 different computers:
Nov 04 2012 19:13 Do I have a startup?

Clay Allsop says having a startup means one of: you've raised money, are bringing in substantial revenue, or have a sizable active user base.

I have a side-project, a web application to manage the academic data for students of speech pathology. I'm a contractor, but in Silicon Valley terms, I'm the tech co-founder and probably CTO.

We've been around since 2005, in one form or another. In about 2009, we spun-off from the institution where the project started, and started taking on paying clients.

Right now, we have 32 institutions that are using the product, with about 8 more in the wings. Each institution has anything from 10 to 100 students. There's our sizable active user base.

We don't have substantial revenue by startup standards , but it is comfortable for a side-project.

We haven't raised, nor attempted to raise, any funding. We're just going by our day-job paychecks, savings, and revenue.

So by Clay's definition, it's a startup. It's not successful (yet), since it can't be all our primary jobs and pay all our salaries.

It can be successful, though, because between the founders we have considerable domain knowledge, technical knowledge, and domain contacts. The business person is a speech pathologist and former professor, which is how this thing got started. Knowing exactly what the customer wants is *gold*.

We don't need to advertise. The user base, speech pathology faculty, are a close-knit bunch (apparently) who love what we have so far. And what we have is a fully functional, post-MVP (Minimum Viable Product, i.e. the least number of features you can release that forms something usable), product.

Do I have a startup? Yes. Is it viable? Probably. Can I get funding? Don't know.

Sep 23 2012 16:12 Migrating a subversion repository to git

I used these instructions to migrate some svn repositories to git:
But I don't have the svn standard layout, with trunk/ branches/ and tags/, so I was able to skip several steps. My condensed flow was:

Retrieve a list of all Subversion committers:

svn log -q | \
awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' \
| sort -u > authors-transform.txt

Edit that file as appropriate.

Clone the Subversion repository using git-svn: Here, I didn't use the --stdlayout argument:

git svn clone [SVN repo URL] --no-metadata -A authors-transform.txt ~/temp

Convert svn:ignore properties to .gitignore: Not required if you didn't have any svn:ignore properties

cd ~/temp
git svn show-ignore > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

Push repository to a bare git repository: I didn't link HEAD to trunk, I linked it to git-svn:

git init --bare ~/new-bare.git
cd ~/new-bare.git
git symbolic-ref HEAD refs/heads/git-svn
cd ~/temp
git remote add bare ~/new-bare.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare

Then delete ~/temp

Rename git-svn to master (instead of renaming trunk):

cd ~/new-bare.git
git branch -m git-svn master

And, since I had no branches, just rename/move new-bare.git to the final resting place, and then clone it out.

Tag: howto