· ten by six retrospective

100 10 1 Retrospective

What’s this again?

1 year and 9 months ago, I started a 100-10-1 project with the goal of coming up with 100 ideas, narrowing them to my 10 favorites, making MVPs out of those, and turning 1 into a finished product. Part of the goal was to blog about the process of building the MVPs and polishing up the final piece.

… Unfortunately I never ended up blogging much beyond the kickoff posts, nor did I finalize any of the MVPs. But I did end up making a few of them. So this post is a retrospective on what I did do, why I didn’t do the rest, and some quick thoughts on the experience overall.

So let’s talk about the projects

These are the 10 projects I ended up choosing.

The four that got made:

And the six that didn’t:

Photo Management Site

This one got the name pic.management. It was almost exclusively worked on while riding bullet trains on a vacation in Japan. Though it’s not often I find myself in that specific environment, it was very productive because the ride was quiet, smooth, and comfortable.

Anyway, much like this report about the project, it got sidetracked. I ended up thinking about the difficult problem of user account controls and permissions instead of anything relating to photos. Somewhat hilariously, I never actually got around to uploading images, and mostly spent my time learning about all the various pieces necessary to keep track of users in a Go based web stack, like cookies, sessions and middleware.

All in all I’d say it was a failure as a project, but a success for learning. I might not use the code I wrote for this project in the future, but knowledge I gained making it has already proved helpful.

Dota 2 api collector / analyzer site

This project is actually two projects in one! The site itself is dotablaze and the api data collector is gotablaze. gotablaze is written in Go, of course, and like most good Go projects, has “go” in the name.

The gotablaze “half” of this project was significantly bigger than the dotablaze part, and was also super super super fun. Figuring out how to use a relatively undocumented api is always an interesting challenge if you’re excited about the result and not pressed to meet a specific deadline.

Some of the most fun parts of this were finding the closest thing to documentation that the Dota 2 api has, playing around with and implementing it, and later needing to figure out what to do about the barely-valid JSON that the api returns: it has keys with the same name in the same scope.

But instead of needing to write my own custom JSON parser, I lucked out because the api happens to also speak XML (being able to use XML is something I never thought I’d be lucky to have). In XML, repeating keys in the same scope is actually how you indicate an array of items.

Another big fun thing for me was storing the data in rethinkdb. Sadly, they’re no longer a company, but I really love their product. It also has a really good 3rd party Go client library.

Part of the goal of using this database was to simply grab the JSON responses and to store diffs of them, made using some suboptimal struct diffing code I wrote. The code doesn’t work on arbitrary structs, though I did initially try and build it that way :) But it was a really good learning experience dealing with something Go doesn’t make easy: defining and diffing arbitrary structures like JSON. Without the use of the wonderful library gojson, all of this would have taken me significantly longer than it already did.

Other small fun backend things about this project include finding out through observation that the Dota 2 api is cached for 15 seconds, so accessing it more frequently than that isn’t necessary. And occasionally the cached output is just “null”, which means you have no updates till the next cache happens 15 seconds later.

Also the api returns JSON and XML pretty printed, with no option to turn that off, which makes the bandwidth usage roughly 36% higher than it really needs to be (at least with the couple sets of JSON files I tested).

The dotablaze half was slightly less fun for me, as I have been much more interested in backend web programming in recent years. But it was a good opportunity to pair with a friend who has the opposite interests, and I certainly learned a lot from him during the process.

Unfortunately, the site is no longer running. It was never anywhere near as good as the UI or data you can get from TrackDota, and revamping it would be too much work. But it was certainly fun while it lasted :)

All in all I’d say this project was a success. Dealing with the Dota 2 api is not easy, but struggling with something and eventually getting what you want out of it can be very rewarding.

Twitter Snowflake Clone

This project ended up being named sanic and I think I enjoyed making it more than any of the other projects.

For those who don’t know (and I didn’t know much earlier than when I started the project), Twitter Snowflake is what Twitter used to use to make all of their t.co short urls, up until 2010. I’m not sure what they use these days, but the general idea back then is that you take all the bits of a timestamp, combine them with bits from a small numeric machine ID, as well as a small numeric data center ID, and smash that all together with a small counter that goes from 0 to 4095. This allows you to use multiple machines in multiple places to generate fairly short IDs, only 10 characters long, that are guaranteed to be unique… for the next 69 years from your starting date. Oh, and you can generate them at theoretically maximum rate of 4,096,000 per second per machine.

I loooove this because it doesn’t solve your problem forever, but it solves it for much longer than you can expect that specific problem to exist. If you think about it, it means they could generate IDs for 16 times longer than the company had even existed at that point!

Anyway, reading their code made this project fairly simple. Their specs also state that they’re looking for at least 10,000 IDs from each process, servable with a fairly low latency. I never got around to grouping and sending the IDs, but when generating the on my late 2013, 2.4 Ghz Core i5 laptop, I got to about 3,850,000 IDs a second, which is pretty close to the maximum.

I’d say this project was a success. It’s pretty simple, like an MVP should be. And it gave me a chance to play around with locks to ensure a user of this library wouldn’t get in trouble using it in a multi-threaded context.

Backblaze B2 api Library

This project was fairly self-contained. It involved implementing Backblaze’s B2 api, which has such amazing documentation that the only effort it really takes is putting in the time. So I did, and modeled some things after this other B2 api client, which taught me a decent amount about good ways to model structs and methods in Go.

I also took the opportunity with this library to really practice unit testing, something I don’t do enough. Though in retrospect, I think it would have been a better situation for integration testing, as running every api endpoint I implemented would probably cost an imperceptible fraction of a cent. Of course I didn’t implement the “large file” api endpoints, which certainly would have helped keep the exhaustive test cost down.

All in all I’d say this project was quite a success. I think I learned the most from it, got some good testing practice in, and at the end of the day have a library that I personally would use to interact with this service. Now I’m not saying you should use it. You should probably use kothar’s, the one I linked above. Mine is not integration tested and hasn’t really been run through its paces.

What happened to the rest?

I had started this project in early 2016. A little later that year I joined the Hillary Clinton campaign. Campaign life is a lot of super intense work with really long hours. It’s not really a job which makes it easy to have side projects. Not because they aren’t allowed, but because the amount of work you’re doing and the amount of time you’re spending doing it is so great that it’s unlikely you have time or energy for other things.

That’s not a hard and fast rule, but it was certainly true for me.

After the campaign, I had to decompress and start my job hunt anew. I had no desire at the time to really do much of anything extracurricular within the programming realm, so I never considered picking back up an experiment with 6 random projects remaining to be completed.

But now that I’ve had enough time off to sit and think, as well as having found gainful employment, it’s definitely time I get back on this horse. That’s actually the major reason I’m writing this. Time to get back into the swing of things.

Anyway, a conclusion

Starting this project was an excellent decision for me. I learned a ton, even though I did less than half of the total work. I definitely recommend it to those in situations where its possible for them.

So will you do it again?

Short answer yes. Longer answer, I think I’m going to try something in the same vein, but ever so slightly different. I’ll be releasing a post on that in the near future.

← Setup Let's Encrypt with NGINX Starting 25:5 →