Dan TannerDan Tanner's mostly technical blog. Any hyperbole expressed here is 100% true. 2024-02-23T00:00:00Zhttps://dantanner.com/Dan Tannerdan@dantanner.comMy indoor cycling setup2024-02-23T00:00:00Zhttps://dantanner.com/post/my-indoor-bike-preferences/<p>This is a list of the gear and software I use for my indoor cycling setup. I'm an old amateur that will ride indoor in
the winter when I'm too lazy to ski or ride my fat bike. It's still a chore for me because I find it pretty
boring, but it's easy to hop on and get a decent workout in without much setup time.</p>
<h2 id="gear" tabindex="-1">Gear <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h2>
<p><img src="https://dantanner.com/indoor-bike/bike-trainer-setup.jpg" alt=""></p>
<table>
<thead>
<tr>
<th style="text-align:left">Thing</th>
<th style="text-align:left">Name</th>
<th style="text-align:right">Price</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">Bike</td>
<td style="text-align:left">My road bike</td>
<td style="text-align:right">-</td>
<td>I don't ride my road bike in the winter so I put that on the trainer.</td>
</tr>
<tr>
<td style="text-align:left">Trainer</td>
<td style="text-align:left"><div style="white-space: nowrap;">Wahoo KICKR</div> Core</td>
<td style="text-align:right">$500</td>
<td>This works great for me and is highly recommended. I've had the Elite Direto and it was a buggy POS. See <a href="https://www.dcrainmaker.com">DC Rainmaker</a> if you want to see other options, but if you can afford it and don't need fancier features, this is the best option for most scenarios IMO. It does require a cassette which is another $50 or so if you don't want to move the existing one on your back wheel.</td>
</tr>
<tr>
<td style="text-align:left">Fan</td>
<td style="text-align:left"><a href="https://www.costco.com/lasko-super-fan-max-multi-purpose-compact-air-mover.product.4000221279.html">Lasko Compact Air Mover</a></td>
<td style="text-align:right">$70</td>
<td>I highly recommend this too. It's small, not too loud, and pushes a strong jet of air in a fairly compact path. I used to use a big metal fan that was the same price but really loud and didn't push nearly as much air.</td>
</tr>
<tr>
<td style="text-align:left">Stand</td>
<td style="text-align:left">whatever</td>
<td style="text-align:right">$50?</td>
<td>I use a <a href="https://www.sweetwater.com/store/detail/MusicStd--on-stage-stands-sm7211b-conductor-stand-with-tripod-folding-base">music stand</a> that works well for this. I use it to hold my laptop (for watching stuff) and phone (for trainer software).</td>
</tr>
<tr>
<td style="text-align:left">Mat</td>
<td style="text-align:left"><a href="https://www.amazon.com/gp/product/B09BQ1FZC8">Cycleclub Bike Mat</a></td>
<td style="text-align:right">$48</td>
<td>This was a well-rated mat on Amazon that works well for me. There are simpler solutions like putting a towel down, but I prefer something sweat proof.</td>
</tr>
<tr>
<td style="text-align:left">Headband</td>
<td style="text-align:left"><a href="https://store.haloheadband.com/Halo-II-pullover-headband-p/mhp.htm">Halo II</a></td>
<td style="text-align:right">$17</td>
<td>I like these. You could use a towel or something else too.</td>
</tr>
<tr>
<td style="text-align:left">Heart monitor</td>
<td style="text-align:left">Wahoo TICKR</td>
<td style="text-align:right">$50</td>
<td>If you have a watch or something else that's fine. I use this because I know it's accurate and it saves my watch battery.</td>
</tr>
</tbody>
</table>
<h2 id="trainer-software" tabindex="-1">Trainer Software <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h2>
<p>tl;dr; I'm using <a href="https://trainerday.com/">Trainer Day</a> and really like it. It's $4/month and I like it better than the
competitors. I watch a movie/TV on my laptop and use my phone for the trainer software, which heavily influences
the trainer software that I like.</p>
<p>This is a personal decision. This is what I want out of training software:</p>
<ul>
<li>Build me a good plan that I can follow</li>
<li>Indoor, I ride solo, so I don't need or want group ride features</li>
<li>Have a good clean user interface that just shows my stats</li>
</ul>
<p>If you're looking for opinions, here's what I've tried and my thoughts on each:</p>
<h4 id="zwift-150-yr-or-15-month" tabindex="-1">Zwift - $150/yr or $15/month <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h4>
<p>The most popular tool today. Group rides/interactions are its main strength, so check it out if that's your jam.
I hate it. I spent about a few dozen hours with it before I couldn't tolerate it anymore.</p>
<ul>
<li>I don't want to stare at shitty cartoon character graphics for an hour or two.</li>
<li>It's buggy.</li>
<li>It chugs through the battery on some devices requiring it to be plugged in</li>
<li>It's kinda pricey</li>
</ul>
<h4 id="rouvy-15-month-for-1-rider-20-for-2-riders-33-for-3-riders" tabindex="-1">Rouvy $15/month for 1 rider, $20 for 2 riders, $33 for 3 riders <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h4>
<p>I tried this after Zwift. The difference is that Rouvy uses real videos of actual rides. This makes it less painful,
but after a half dozen rides, I concluded that trainer software that I'm supposed to focus on during the ride just is
not my jam.</p>
<h4 id="trainer-road-190-yr-or-20-month" tabindex="-1">Trainer Road - $190/yr or $20/month <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h4>
<p>It's marketed for the hardcore crowd. I used this a few years ago when I was riding more seriously and liked it.
It just gives you a screen with your stats, which is what I prefer.
It has a bunch of AI features, which is code for "this seems like a good way to justify charging $20/month".
But Trainer Day fits my needs just as well.</p>
<h4 id="indievelo-free-beta-testing-at-time-of-writing" tabindex="-1">indieVelo - free beta testing at time of writing <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h4>
<p>I didn't try this so don't have an opinion, but it's another option.</p>
<h4 id="trainer-day-4-month" tabindex="-1">Trainer Day - $4/month <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h4>
<p>I really like this software. It's ergonomic, concise, stable, and you can tell that it's built with love.
I also like that it remains tightly focused on its features which align well with what I want.</p>
<h2 id="tips" tabindex="-1">Tips <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h2>
<p>If you're new riding on a trainer, one bit of advice is that it's just not as comfortable as riding for real on your
bum. To mitigate this, stand every few minutes or so to keep from getting numb and uncomfortable.</p>
<h2 id="training-plans" tabindex="-1">Training Plans <a class="header-anchor" href="https://dantanner.com/post/my-indoor-bike-preferences/">#</a></h2>
<p>Don't look to me for professional advice, but there's one thing I want to mention - some modern training plans are
gentler but just as effective. That's one important criticism of many plans out there - the training plan rides are
considered by some experts to be too intense on average. The resource I've taken advice from lately are videos on
polarized training from Dylan Johnson like <a href="https://www.youtube.com/watch?v=Ju3McjlSoAg">this one</a> that recommend most
rides should be at an easy pace (like, actually easy). I'm enjoying trying this technique so far for a couple of
reasons:</p>
<ul>
<li>makes me feel better because I have more energy after the ride for other stuff I want to do</li>
<li>makes me feel better because trusting the science, I can feel more at ease knowing this will make me faster and avoid
burnout</li>
</ul>
<p>My advice is to be aware of this as you're building your training plan. Some training plan software is still using the
old school technique of <code>more suffer = more gain</code>, which can be counter-productive and less fun. Hope this helps.</p>
Spring Rites2024-02-19T00:00:00Zhttps://dantanner.com/post/spring-rites/<h2 id="a-caution-against-annotation-based-web-frameworks" tabindex="-1">A caution against annotation-based web frameworks <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h2>
<p><img src="https://dantanner.com/annotation/spring-rites.jpg" alt=""></p>
<!-- TOC -->
<ul>
<li><a href="https://dantanner.com/post/spring-rites/">Context</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">Original problems solved by the Spring Framework</a>
<ul>
<li><a href="https://dantanner.com/post/spring-rites/">"Helps me configure my application"</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">"Reduces application server vendor lock-in"</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">"Helps me manage my large monolithic app"</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">"Reduces dependency management issues"</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">"Provides a productive MVC pattern"</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">"Helps me package my application more easily"</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">"Helps me unit test my code more easily"</a></li>
</ul>
</li>
<li><a href="https://dantanner.com/post/spring-rites/">Starting from scratch</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">Why are people still using Spring?</a>
<ul>
<li><a href="https://dantanner.com/post/spring-rites/">On starting projects quickly</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">On people and the sunk cost fallacy</a></li>
</ul>
</li>
<li><a href="https://dantanner.com/post/spring-rites/">On Micronaut</a></li>
<li><a href="https://dantanner.com/post/spring-rites/">On the future of the JVM</a></li>
</ul>
<!-- TOC -->
<h2 id="context" tabindex="-1">Context <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h2>
<p>My current job is to help developers write better software at a large retail company. There's a few thousand of them,
so there are a lot of opinions and techniques used. Most of the business application software is written on the
JVM, and I spend a lot of my time on documentation and reference material for best practices on how to build high
quality software within our ecosystem. "High quality" meaning performant, maintainable, secure, efficient, concise,
etc...</p>
<p>I currently recommend building new projects on the JVM using a "vanilla" Kotlin approach. By vanilla, I mean don't use
annotation-based application frameworks like Spring or Micronaut. This concept of not using an application framework for
a Java project is foreign and unpopular to some people. Usually <em>foreign</em> to people unfamiliar with development in other
languages, and <em>unpopular</em> typically with people that have only ever used an application framework.</p>
<p>What makes my job more difficult is that I'm going against the grain of popularity. Applications built using Spring are
the most popular stack at my company (over half). So in conversations with people about this choice, I'm usually
in the situation of defending my viewpoint for this "new" technique of not using a framework, and trying to convince
people to move from a technique they're used and know works, to a new technique they usually have limited experience
with.</p>
<p>In this article, I'm not going to defend my viewpoint. I'm going to attack the viewpoint that using an annotation-based
framework like Spring Boot or Micronaut makes sense in 2024.
Sure, it "works", and that's the most common defense I see in forums. Carburetors also still work, but you don't see
them in modern engines.</p>
<p>The theme of my criticisms throughout this article are generally this:</p>
<ol>
<li><strong>Using annotations for complex features are inherently more difficult to test, maintain, and extend compared to its
vanilla Kotlin counterpart.</strong></li>
<li><strong>Large application frameworks like Spring are now the cumbersome things they originally displaced.</strong></li>
<li><strong>Annotations make it harder to reason about your code.</strong></li>
</ol>
<h2 id="original-problems-solved-by-the-spring-framework" tabindex="-1">Original problems solved by the Spring Framework <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h2>
<p>Remember that Spring was created as a <strong>lighter weight alternative</strong> to the bloated, immature application servers at the
time. Think Entity Beans, vendor-driven application servers, and other garbage that got in the way. This was before
technology like Ruby on Rails, Django, JSON, Redis, Elasticsearch, NGINX, AWS, GCP, SSDs, and so many other things that
have evolved our options.</p>
<p>Here's the original category of features the Spring Framework advertised and delivered on in 2003:</p>
<ul>
<li>Helps me configure my application</li>
<li>Reduces application server vendor lock-in</li>
<li>Helps me manage my large monolithic app</li>
<li>Reduces dependency management issues</li>
<li>Provides a productive MVC pattern</li>
<li>Helps me package my application more easily</li>
<li>Helps me unit test my code more easily</li>
</ul>
<p>These were new, <em>real problems</em> <strong>back then</strong>.</p>
<p>Not today though. These features are either irrelevant today, or are actually made worse by using Spring Boot.</p>
<h3 id="helps-me-configure-my-application" tabindex="-1">"Helps me configure my application" <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h3>
<p>tl;dr; Use <a href="https://github.com/sksamuel/hoplite">hoplite</a> to load your configuration into type-safe data classes.
For example, this code will load configuration from various sources in order, populating an arbitrary data class that
you define:</p>
<pre><code class="language-kotlin">import com.sksamuel.hoplite.ConfigLoaderBuilder
class App {
private val config = ConfigLoaderBuilder.default()
.addFileSource("/etc/myapp/dev.conf", optional = true)
.addResourceSource("/default.conf")
.build()
.loadConfigOrThrow<Config>()
private val db = Db(config.db)
// etc...
}
</code></pre>
<p>Where your <code>Config</code> class might look like this:</p>
<pre><code class="language-kotlin">data class Config(
val http: HttpConfig,
val metrics: MetricsPublisher.Config,
val db: DbConfig,
)
data class HttpConfig(
val server: HttpServerConfig,
val client: ClientConfig,
)
// etc
</code></pre>
<p>There are a bunch more features you can use, but here's an example of a startup log using hoplite:</p>
<pre><code class="language-shell">Property sources (highest to lowest priority):
- Env Var Overrides
- System Properties
- /Users/dan/.userconfig.<ext>
- $XDG_CONFIG_HOME/hoplite.<ext>
- classpath:/local.conf
- classpath:/default.conf
Used keys: 4
+-----------------------------------+-------------------------+
| Key | Source |
+-----------------------------------+-------------------------+
| db.databaseName | classpath:/default.conf |
| db.password | classpath:/default.conf |
| http.client.item.cache.cacheSize | classpath:/default.conf |
+-----------------------------------+-------------------------+
Unused keys: 1
+--------------------------+-------------------------+
| Key | Source |
+--------------------------+-------------------------+
| metrics.tags.team | classpath:/default.conf |
+--------------------------+-------------------------+
--End Hoplite Config Report--
</code></pre>
<p>This is out of the box functionality, and it was about 30 lines of code to configure.</p>
<p>Instead, what do you get with Spring?</p>
<p><strong>Quick</strong>, tell me what each of these annotations mean and how they should be used when you need to configure a custom
bean for your application:</p>
<ul>
<li><code>@Configuration</code></li>
<li><code>@ConfigurationProperties(prefix = "someprefix")</code></li>
<li><code>@ConstructorBinding</code></li>
<li><code>@EnableConfigurationProperties(SomeConfig::class)</code></li>
</ul>
<p>Did you know the correct answer, which is that you need to combine those annotations in pairs like so?:</p>
<pre><code class="language-kotlin">@Configuration
@ConfigurationProperties(prefix = "someprefix")
data class MetricsConfig(
val config: MetricsPublisher.Config,
)
@ConstructorBinding
@EnableConfigurationProperties(SomeConfig::class)
class Configuration(
val metricsConfig: MetricsConfig,
)
</code></pre>
<p>Just kidding - the correct answer is a different order of the annotations!</p>
<p><img src="https://dantanner.com/annotation/wizard-annotation.jpeg" alt=""></p>
<p>How would you know though? And how long would it take for someone new to find the correct answer?</p>
<h3 id="reduces-application-server-vendor-lock-in" tabindex="-1">"Reduces application server vendor lock-in" <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h3>
<p>If you're referring to the ability to choose the underlying server engine, this is a common feature for all major
HTTP server libraries today.
e.g. <a href="https://ktor.io/docs/engines.html">Ktor</a>, <a href="https://www.http4k.org/guide/reference/servers/">http4k</a>. (<em>http4k even
lets you use a Ktor engine, even though they're competitors!</em>)</p>
<p>The real lock-in is the application server or library interface. For example, you can't switch from Spring to
Micronaut without a rewrite of your HTTP handling code. To this point, Spring is just another vendor.</p>
<h3 id="helps-me-manage-my-large-monolithic-app" tabindex="-1">"Helps me manage my large monolithic app" <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h3>
<p>I went from using Inversion of Control frameworks, to <a href="https://github.com/google/guice">Guice</a> as a lighter alternative,
to what I do which is just new up each singleton in the application class. It's not problematic, and it's much easier
to reason about. Maybe someday we'll all be using <a href="https://www.youtube.com/watch?v=GISPalIVdQY">context receivers</a>, but
for now direct dependency injection from the main class is the simplest solution that works for me.</p>
<p>It's pretty similar
to <a href="https://grafana.com/blog/2024/02/09/how-i-write-http-services-in-go-after-13-years/">this recent post on how one person writes HTTP services in Go</a>,
which uses test-friendly techniques like functions as arguments to services.</p>
<p>That said, who's building large monolithic apps these days? The current standard for most web apps is smaller, more
granular apps, which means less dependencies. Most applications I write these days have a 100 line long <code>App</code> class
that loads the configuration, creates the services and routes, and starts the services from a main method. It's simply
not a real problem any longer for most situations.</p>
<h3 id="reduces-dependency-management-issues" tabindex="-1">"Reduces dependency management issues" <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h3>
<p>Raise your hand if you or someone you know has a Spring application stuck using old versions because they don't have the
time to migrate it to the latest version.</p>
<p>Keep your hand raised if you've been flagged by security for running with a high severity vulnerability in your app, but
can't easily fix the issue because the vulnerable version is a required dependency of your application framework.</p>
<p>Want to know who almost never have that problem? People that don't use an application framework, and just use libraries.</p>
<p>The best way to do it today is <a href="https://docs.gradle.org/current/userguide/platforms.html">Gradle catalogs</a> + some
automated
dependency update tool like <a href="https://www.mend.io/renovate/">renovate</a>
or <a href="https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/about-dependabot-version-updates">dependabot</a>.
For example, I use renovate at work, and my configuration looks like this:</p>
<pre><code class="language-json">{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:best-practices",
"group:monorepos",
"group:recommended",
"group:allNonMajor",
":enablePreCommit"
],
"schedule": [
"every weekday",
"every weekend"
],
"timezone": "America/Chicago",
"semanticCommits": "enabled",
"dependencyDashboard": true,
"dependencyDashboardAutoclose": false,
"dependencyDashboardApproval": false
}
</code></pre>
<p>Each week I get an automated PR for each app that bumps all the minor versions in one PR, and creates a separate PR
for any major (potentially breaking) dependency updates.</p>
<p>Know what I'm not doing? I'm
not <a href="https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.0-Migration-Guide">reading the Spring Boot migration guide</a>.</p>
<p>And more on dependency management and security...do you know how many lines of third-party code is running in production
for your little Spring Boot app?</p>
<p>In an A/B comparison between a Spring Boot reference application and its identically-featured vanilla Kotlin equivalent,
the Spring Boot application had an average of 162 runtime dependencies (JARs) vs 61 for vanilla Kotlin. That's a 162%
increase in runtime dependencies, and represented about 2 million lines of running code for Spring Boot vs about 800k
for the Kotlin equivalent. That's a lot of potentially vulnerable code that you don't have control over.</p>
<h3 id="provides-a-productive-mvc-pattern" tabindex="-1">"Provides a productive MVC pattern" <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h3>
<p>Using <a href="https://www.http4k.org/guide/howto/use_a_server_backend/">http4k</a> as an example of how easy it is today with
modern libraries:</p>
<pre><code class="language-kotlin">fun main() {
val app = { request: Request -> Response(OK).body("Hello, ${request.query("name")}!") }
val server = app.asServer(Undertow(9000)).start()
}
</code></pre>
<p>The <code>app</code> function can even be unit tested without starting a server!</p>
<h3 id="helps-me-package-my-application-more-easily" tabindex="-1">"Helps me package my application more easily" <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h3>
<p>Another thing that is trivial these days using
the <a href="https://docs.gradle.org/current/userguide/application_plugin.html">gradle application plugin</a>.</p>
<p>Here's an example of my gradle file that configures the application plugin, which will create an executable directory
structure for my application:</p>
<pre><code class="language-kotlin">plugins {
application
}
application {
applicationName = "app"
mainClass.set("com.example.api.AppKt")
}
</code></pre>
<p>Then run <code>./gradlew distTar</code> as part of the CI build process. This process is identical for any application built
these days regardless of whether you're using an application framework or not.</p>
<h3 id="helps-me-unit-test-my-code-more-easily" tabindex="-1">"Helps me unit test my code more easily" <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h3>
<p>Spring originally made testing easier, but the shift to annotations has essentially eliminated the ability to unit test
code.</p>
<p>The reason is that you can't unit test a class with annotations like <code>@GET</code> or <code>@Transactional</code> in it.</p>
<p>Everything's an integration test when you need to spin up the framework context, which is slower and more complex than
unit testing.</p>
<p>In the same A/B comparison of identically featured applications built using Spring Boot and vanilla
Kotlin, the mean test execution time was over 185% slower for Spring Boot versus vanilla Kotlin.
(16 seconds for vanilla Kotlin, 46 seconds for Spring Boot on an M1 Max MacBook Pro)</p>
<h2 id="starting-from-scratch" tabindex="-1">Starting from scratch <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h2>
<p>Picture this scenario: You're a good, experienced programmer in other languages, but you're new to the JVM,
and you've been tasked to determine the stack for a new JVM application.</p>
<p>This application will serve up API requests using Undertow, make HTTP client calls using OkHttp, and consume messages
using Apache Kafka. It will use these libraries regardless of whether you use Spring or not.</p>
<p>Would you rather learn <a href="https://kafka.apache.org/documentation/#consumerconfigs">how to use Kafka</a> directly,
or would you rather learn how to use Kafka
through <a href="https://docs.spring.io/spring-kafka/reference/kafka/receiving-messages/message-listeners.html">Spring's abstraction</a>?</p>
<p>If you're not already familiar with the contents of those two links, please just click into those and briefly scan
them. The tl;dr; is that Kafka is a very powerful and useful messaging tool, but there are a lot of dials
you need to adjust to use it correctly.</p>
<p>Using the default Kafka <code>@KafkaListener</code> annotation and wondering why your consumer performance is over an order of
magnitude slower than a plain consumer?
Oh - it was committing offsets on every message? Aw shucks, guess you picked the wrong message listener interface out of
the eight possible methods.</p>
<p>And lost some messages because your consumer was set to autocommit? Unfortunately you now have to not only read the
Kafka
docs, but also read how that relates to Spring's abstraction over it.</p>
<p>This pattern is true for every library used in the application. You don't have less to understand - you have <em>more</em> to
understand!</p>
<ul>
<li>Spring + OkHttp</li>
<li>Spring + Undertow</li>
<li>Spring + Kafka</li>
<li>Spring + Jackson</li>
<li>Spring + OpenTelemetry
and so on...</li>
</ul>
<p>This also assumes Spring gives you the necessary control over the tool you need, which isn't always the case.</p>
<p>Does that sound appealing?</p>
<h2 id="why-are-people-still-using-spring" tabindex="-1">Why are people still using Spring? <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h2>
<p>Because <strong>Spring makes it easy to start a project</strong>, and <strong>most people advocating Spring don't realize that what they
consider a sunk cost is really a significant fixed cost</strong>.</p>
<h4 id="on-starting-projects-quickly" tabindex="-1">On starting projects quickly <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h4>
<p>Tools like the <a href="https://start.spring.io/">Spring Initializr</a> and <a href="https://micronaut.io/launch/">Micronaut Launch</a> make
it easier to start a project. This is a real benefit.
But just about everyone else makes starting a project easier now too. e.g.
the <a href="https://start.ktor.io">Ktor Project Generator</a> and <a href="https://toolbox.http4k.org/">http4k's toolbox</a>.</p>
<p>To Spring's credit, the documentation is very thorough and generally better than most other projects.
This gets you going. When things get tough, and you need to step through code to see what's really going on, this is
where annotations work against you. You end up poring over heaps of documentation and obtuse stack traces spending
hours or days trying to figure out what's wrong with the system.</p>
<h4 id="on-people-and-the-sunk-cost-fallacy" tabindex="-1">On people and the sunk cost fallacy <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h4>
<p>People use annotation-based frameworks because they work. Many projects have been built using them, and people are
hesitant to change from what they know works.</p>
<p>As an example of what not to do, consider the frontend development landscape over the last decade. The amount of
churn and short-lived fads is embarrassing, even for our immature industry. And that debate is far from settled yet.</p>
<p>Some JVM developers in the backend community have settled though. I'm one of the first people to say things
like "let's make a choice to use boring technology X for Y years", but the time to continue using annotation-based
frameworks like Spring Boot has passed.</p>
<p>For those that have given the vanilla Kotlin approach a try, the response has been overwhelmingly positive
in my observations. They typically become strong advocates of it once they've spent a couple of weeks with it.</p>
<p><img src="https://dantanner.com/annotation/annotation-venn.svg" alt=""></p>
<p>The reason people using a vanilla Kotlin approach are often joyful is because they've stopped paying the framework tax.
People that have only used heavy frameworks don't know that these constant burdens can be lifted:</p>
<ul>
<li>Being able to reason about the application more clearly when there are no magic annotations</li>
<li>Being able to more easily see and control how the application is configured</li>
<li>Being able to unit test routes, services, and message processing without starting a container or application context</li>
<li>Being able to reliably upgrade dependencies in a matter of seconds, not hours or days</li>
</ul>
<h2 id="on-micronaut" tabindex="-1">On Micronaut <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h2>
<p>A relatively small percentage of people are tired of Spring and have embraced Micronaut.</p>
<p>Micronaut in my opinion is just a slightly less-bad version of Spring Boot though, and solves the wrong problems.</p>
<table>
<thead>
<tr>
<th>Advertised Feature</th>
<th>My viewpoint</th>
</tr>
</thead>
<tbody>
<tr>
<td>Polyglot Framework</td>
<td>This should be listed as a limitation, not an advertised feature. If you don't use an annotation-based inversion of control framework, you can use whatever language you want without limitation.</td>
</tr>
<tr>
<td><div style="white-space: nowrap;">Natively cloud-native</div></td>
<td><em>Distributed tracing?</em> Use OpenTelemetry. <em>Cloud runtimes?</em> Use docker or any of the available hosting services. <em>Discovery services?</em> Use your infrastructure's routing features natively.</td>
</tr>
<tr>
<td>Fast data-access config</td>
<td>Use <a href="https://github.com/sksamuel/hoplite">hoplite</a>, the repository pattern, and your data access library directly.</td>
</tr>
<tr>
<td>Smooth learning curve</td>
<td>The guide alone is over 25,000 lines for your smooth reading pleasure. Do you consider learning about Bean qualifiers and Scopes on meta annotations to be essential complexity?</td>
</tr>
<tr>
<td>Fast, easy unit testing</td>
<td>Just like <a href="https://dantanner.com/post/spring-rites/">Spring Boot</a>.</td>
</tr>
<tr>
<td>Aspect-oriented API</td>
<td>You're gonna have to debug your application when things go wrong, yes? Are you comfortable with stack traces coming from <a href="https://docs.micronaut.io/latest/guide/index.html#aop">this</a>? There are words like "simply" and "trivial" in there that are simply not.</td>
</tr>
<tr>
<td>Seamless API Visibility</td>
<td>Generated swagger specs is a nice feature and not enough people do that IMO. It's available in <a href="https://www.http4k.org/guide/howto/create_a_swagger_ui/">lightweight tools though</a>.</td>
</tr>
<tr>
<td>AOT Compilation</td>
<td>Micronaut's startup time and memory footprint is much smaller than Spring's, but so is not using a framework at all. Also, most people don't care about a 10 second server startup time unless you're writing functions as a service. And if memory consumption is a barrier, you probably aren't using Java.</td>
</tr>
</tbody>
</table>
<p>Micronaut suffers from most of the same problems Spring Boot has, but is less popular, and isn't a very compelling
reason for those using Spring to switch to it in my opinion.</p>
<h2 id="on-the-future-of-the-jvm" tabindex="-1">On the future of the JVM <a class="header-anchor" href="https://dantanner.com/post/spring-rites/">#</a></h2>
<p>Java is still widely used, but it's on the decline. Competition comes from languages like Python, Go, C#, and
JavaScript. <em>Especially</em> Python being more approachable than any other mainstream language.</p>
<p>These languages don't use annotation-based inversion of control frameworks, and programmers of those languages aren't
asking for it. They view annotations as an annoying obstacle that gets in the way of onboarding and productivity.
I've polled them in their communities; this is the #1 reason they're repelled by Java.</p>
<p>And this bums me out. I've spent over half of my 27-year career writing for the JVM, and <em>I</em> don't want to code
in Java either. But <a href="https://dantanner.com/post/kotlin-v-java/">Kotlin is great</a>.
It's my favorite language on the JVM above Java, Groovy, Scala, and Clojure.
Kotlin is better in many ways than any other language for typical use cases of server applications. It still carries some
of the baggage of Java's history, but it's a <em>really</em> good language.</p>
<p>I've taught a Python programmer Kotlin and watched them grow to really like it.
That doesn't happen with annotation-based frameworks; it's the opposite in my experience, and I think the continued
use of annotation-based frameworks will continue the decline of Java.</p>
<p>Also consider how tools like ChatGPT and Copilot will affect the situation.</p>
<p>Especially for newer programmers, these tools generate code the person doesn't fully understand. Languages like Python
and Go have been built with an eye toward approachability, so will be readable in more situations compared to languages with
a richer syntax like Java. But have Copilot emit a barrage of annotations? Good luck with that. These developers will
suffer, and so will your project.</p>
<p>It doesn't have to be like that though. Use Kotlin, ditch the annotation framework, and use a simpler method.</p>
Kotlin vs Java in 20212021-10-26T00:00:00Zhttps://dantanner.com/post/kotlin-v-java/<p>Since the arrival of Java 17, I've heard a few people wonder "Does Kotlin still make sense, or should I just use Java?"</p>
<p>My opinion: Kotlin is still a far better language, and some of its critical design features will never be matched in
Java.</p>
<h2 id="null-safety-and-immutability" tabindex="-1">Null Safety and Immutability <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h2>
<p>Kotlin lets you write elegant null-safe code, and forces you to be explicit about null-safety.</p>
<p>For example, consider this data class:</p>
<pre><code class="language-kotlin">data class Person(val name: String)
</code></pre>
<p>Because of the <code>val</code> keyword, you can't forget to set name when instantiating a Person, and you can't set it to null later.
That's an entire class of nasty bugs wiped out with a single elegant language design.</p>
<p>Conversely, when something <em>can</em> be null, the compiler forces you to explicitly handle it in a concise manner, also
giving you an escape hatch if you choose to be unsafe about it.</p>
<pre><code class="language-kotlin">var b: String? = "abc"
b.length // compile error: variable 'b' can be null
// you must use a null-safe operator
b?.length // which evaluates to null if 'b' is null, or the length of 'b' if it is not null
// or you can add the `!!` operator to a call if you're willing to accept the possibility of a null pointer situation.
b!!.length
</code></pre>
<p>Java bolted on the <code>Optional</code> container to try to solve this problem, but it's clumsy and rarely used in the code I've
seen. That's a recurring theme in many of the features; Kotlin makes it easy to write concise and stable code.
This one's just really really important in my opinion, and is worth the price of admission on its own.</p>
<p><em><strong>You could stop reading this article now and that's enough reason alone to use Kotlin in my opinion.</strong></em></p>
<p>Official docs on null safety <a href="https://kotlinlang.org/docs/null-safety.html">here</a>.</p>
<h2 id="some-of-my-favorite-features" tabindex="-1">Some of my favorite features <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h2>
<p>Now for some reasons I really enjoy writing Kotlin.
There are plenty of other features, but these are the favorites that come to mind.</p>
<h3 id="collection-functions" tabindex="-1">Collection functions <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h3>
<p>There's a <em>ton</em> of them, and they're powerful and pragmatic.
Full docs <a href="https://kotlinlang.org/docs/collections-overview.html">here</a>; some commonly used examples:</p>
<pre><code class="language-kotlin">val numbersGreaterThanOne = listOf(1, 2).filter { it > 1 } // [2]
val stringLengths = listOf("bear", "cat").map { it.length } // [4, 3]
val firstTwoElements = listOf(2, 4, 6).take(2) // [2, 4]
val sum = listOf(1, 5).sumOf { it } // 6
</code></pre>
<p>Java has streams, but its a small fraction of what's available in Kotlin, and is again a bolt-on rather than
integral part of the language. Java has dozens; Kotlin has hundreds.</p>
<h3 id="extension-functions" tabindex="-1">Extension functions <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h3>
<p>From the <a href="https://kotlinlang.org/docs/extensions.html">docs</a>:</p>
<blockquote>
<p>Kotlin provides the ability to extend a class with new functionality without having to inherit from the class or
use design patterns such as Decorator. This is done via special declarations called extensions.</p>
</blockquote>
<p>For example, add a utility function to Date that converts it to a UTC ZonedDateTime:</p>
<pre><code class="language-kotlin">fun Date.toUtcZonedDateTime(): ZonedDateTime = ZonedDateTime.ofInstant(this.toInstant(), ZoneOffset.UTC)
val date = Date()
val zonedDateTime = date.toUtcZonedDateTime()
</code></pre>
<p>Another example, encapsulating a standard choice for a hashing algorithm so that usage is concise and
easily refactorable:</p>
<pre><code class="language-kotlin">import com.google.common.hash.Hashing
fun ByteArray.hash(): ByteArray = Hashing.murmur3_128().hashBytes(this).asBytes()
val hashedBytes = "abc".toByteArray().hash()
</code></pre>
<p>Once you start using them, you'll find all sorts of ways to make your code cleaner. You can create DSLs effortlessly,
and add missing features to code while keeping it readable and easily testable.</p>
<h3 id="default-and-named-arguments" tabindex="-1">Default and named arguments <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h3>
<p>Default arguments are all about more compact, readable, and refactorable code.
Named arguments give you readability when you want, and also reduce bugs in the case of multiple
arguments with the same type.</p>
<pre><code class="language-kotlin">fun makeHttpCall(url: String, timeoutSeconds: Int = 10, retries: Int = 3) { ... }
// all of these work
makeHttpCall(url = "http://foo.com", timeoutSeconds = 30, retries = 10)
makeHttpCall(url = "http://foo.com", timeoutSeconds = 30)
makeHttpCall(url = "http://foo.com", retries = 0)
makeHttpCall(url = "http://foo.com")
</code></pre>
<p>With Java, you still need gobs of overloads or builders, and even then without named arguments usage gets tricky.</p>
<pre><code class="language-java">public void makeHttpCall(String url) {
makeHttpCall(url, 30);
}
public void makeHttpCall(String url, int timeoutSeconds) {
makeHttpCall(url, timeoutSeconds, 3);
}
public void makeHttpCall(String url, int timeoutSeconds, int retries) {
// ...
}
makeHttpCall("http://foo.com", 30, 10);
// which arg is the timeout? hope i didn't screw up.
makeHttpCall("http://foo.com", 10);
makeHttpCall("http://foo.com");
// how do we call with just a url and retries?
// we'd need a new method that doesn't conflict with the
// existing method that takes a String and int.
</code></pre>
<p>Docs <a href="https://kotlinlang.org/docs/functions.html#named-arguments">here</a></p>
<h3 id="string-interpolation" tabindex="-1">String interpolation <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h3>
<pre><code class="language-kotlin">val score = 1
println("the score is $score. The score minus one is ${score - 1}")
</code></pre>
<p>This is the cleanest string interpolation technique in my opinion.</p>
<h3 id="data-classes-and-copy" tabindex="-1">Data classes and .copy() <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h3>
<p>Java finally has records now. Something I use all the time though, especially when combined with
immutable data structures, is the <code>copy</code> function. For example a common test data setup
pattern looks something like this:</p>
<pre><code class="language-kotlin">// in Person.kt
data class Person(val name: String, age: Int)
// in PersonTest.kt
fun randomPerson(): Person {
return Person(name = someRandomString, age = someRandomAge)
}
@Test fun testPerson() {
val bob = randomPerson().copy(name = "bob")
}
</code></pre>
<h3 id="function-types-as-class-parameters" tabindex="-1">Function types as class parameters <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h3>
<p>This is a really powerful and flexible capability. It does come at a slight cost in clarity, and IDE support is
limited, but is preferred by myself and many others I've worked with.
First off, what's a function type? Docs <a href="https://kotlinlang.org/docs/lambdas.html#function-types">here</a>, but the
most common form I use looks like this:</p>
<p><code>(Int) -> String</code></p>
<p>which can be broken down like this:<br>
<img src="https://dantanner.com/kotlin-2021/function-type.png" alt=""></p>
<p>An example function type that takes two Int parameters and returns a String would be:
<code>(Int, Int) -> String</code></p>
<p>A function with no parameters that returns a String would be:
<code>() -> String</code></p>
<p>A function that doesn't return a useful value must specify <code>Unit</code>. e.g.:
<code>(Int) -> Unit</code></p>
<p>OK great - you can define functions that have a signature but don't have a name. What can we do with that?
How about pure dependency injection without the need for mocks?</p>
<pre><code class="language-kotlin">// a "real" DataRepository that saves data to a repository
class DataRepository {
fun saveStatus(status: String): Boolean {
// save the name to the database and return true if it was updated
}
}
// a "real" Publisher
class Publisher {
fun publishStatus(status: String) {
// publish the status via some integration
}
}
// a service that performs complex logic that needs to be unit tested
class FooService(
saveStatus: (String) -> Boolean,
publishStatus: (String) -> Unit
) {
fun handleStatus(status: String) {
saveStatus(status)
publishStatus(status)
}
}
// the class wiring up the real stuff together
class Application {
private val dataRepository = DataRepository()
private val publisher = Publisher()
private val fooService = FooService(
saveStatus = dataRepository::saveStatus,
publishStatus = publisher::publishStatus
)
// whatever else the application needs to do to get wired up and run
}
// the service unit tests
class FooServiceTest {
@Test fun `everything works great`() {
val fooService = FooService(
// set the saveStatus parameter to a lambda that ignores the String parameter and returns true
saveStatus = { _ -> true },
// set the publishStatus parameter to a lambda that ignores everything and returns Unit implicitly
publishStatus = { }
)
fooService.handleStatus("yay") // along with whatever assertions you want
}
@Test fun `saveStatus failure should do the right thing`() {
val fooService = FooService(
// set the saveStatus parameter to a lambda that blows up, simulating a problem in the repository
saveStatus = { _ -> throw RuntimeException("the sky is falling") },
// set the publishStatus parameter to a lambda that ignores everything and returns Unit implicitly
publishStatus = { }
)
// shouldThrow is an example feature of the fantastic kotest assertion library
// i.e. we've injected behavior into the service indicating a failure should happen
// and we should handle it properly
shouldThrow<RuntimeException> {
fooService.handleStatus("should fail")
}
}
}
</code></pre>
<p>No DI frameworks needed. No mocking needed. Just pure function signatures with full precise control of the contract
behaviors you're building and testing.
This works great for situations like this where you need to wire up a single production implementation and want one
or more test implementations. The compiler will keep the code honest. But, one caveat is that the <code>Find Usages</code>
functionality of the IDE doesn't work because the functions are too ambiguous for the current versions of IntelliJ.
That does add a little more mental overhead to maintenance, along with the mental overhead of the function type as a
parameter itself. But in practice this hasn't been an issue on my projects, and the benefits outweigh the drawbacks
in my opinion.</p>
<p>All that said, interfaces and regular old functions obviously still have their place and should be used where
appropriate.</p>
<h3 id="lots-more-common-idioms" tabindex="-1">Lots more common idioms <a class="header-anchor" href="https://dantanner.com/post/kotlin-v-java/">#</a></h3>
<p>The Kotlin docs maintain a list of frequenty used idioms <a href="https://kotlinlang.org/docs/idioms.html">here</a>.
It's a great place for practical examples of good code.</p>
50 Million Essential Homeowner Tools2021-02-06T00:00:00Zhttps://dantanner.com/post/essential-homeowner-tools/<h2 id="overview" tabindex="-1">Overview <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<p>This article summarizes what I think of as essential homeowner tools for people interested in basic maintenance and improvements. With the amount of information available online today, it's easier than ever to learn. For example, I love <a href="https://www.youtube.com/channel/UC2rzsm1Qi6N1X-wuOg_p0Ng">Project Farm</a>. Unfortunately there's also a lot of incomplete and just plain bad information out there.</p>
<p>So this my attempt to condense twenty years of home ownership experience into a concise but complete starting point. I made it by walking around my garage and basement, taking pictures of all my tools, then narrowed it down to the things I consider essential. Then I showed this list to a few friends, and they gave me some more great ideas.</p>
<p>This is by no means a complete list. A person's list of tools will grow over time as they tackle more specific and larger projects. For example, I have a cordless nail gun that I really like, but it's not on this list. I used it when I installed trim in my basement remodel. I don't use it often though, and don't consider it an essential tool that everyone needs.</p>
<p>I've also tried to categorize tools. This is useful for organization and what interests you. For example, tiling a kitchen backsplash is easier than you might think, but mudding drywall is really hard to do well without experience.</p>
<h2 id="what-you-don-t-need" tabindex="-1">What You Don't Need <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<p>I mention this up front because in a perfect world, we'd be able to look for a "high quality home tool kit" online or at a local hardware store, and be presented with some good options. Unfortunately I have never seen a decent home tool kit. They're cheap and low quality. Only buy one if you can't afford to get better tools, and then replace them as you can over time. On the other hand, don't waste your money with a 10-piece cordless combo power tool kit. While it's useful to have a single type of battery for multiple tools, you should spend your money on tools you'll actually use.</p>
<h2 id="brands-and-quality" tabindex="-1">Brands and Quality <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<p>You can roughly put tool brands into two categories of quality and price. Cheap and mediocre quality, or more expensive and good quality. If you're going to use a tool a lot, buy a good one. If you have a project where you only need to use a tool once and it doesn't have to be that good, then it's ok to use a cheaper one. There's often a big difference in quality and usability between a $100 and $150 tool.</p>
<h2 id="the-essentials" tabindex="-1">The Essentials <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<p>You'll probably use all of these within your first year of home ownership.</p>
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/tape-measure.png" alt=""></td>
<td><strong>Tape Measure</strong></td>
<td>Start with one, and buy another when you can't remember where the first one went. There's a particular brand and model that I really like - the FastCap Standard Reverse 25'. It's called reverse because its measurements are written in both directions, so you never have to read the numbers upside-down. It has other some nice features built-in like an erasable notepad, better belt clip, and a pencil sharpener.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/screwdrivers.png" alt=""></td>
<td><strong>Screw Drivers</strong></td>
<td>Some tools come in assortments of essential sizes that are cheaper to buy in sets. Screwdrivers are one of them. A 6-piece or larger set containing Standard and Slot heads in common sizes is a good starting point.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/cordless-drill.png" alt=""></td>
<td><strong>Cordless Drill/Driver</strong></td>
<td>Everyone needs a general purpose cordless drill/driver. I recommend getting a Ridgid 18V for this tool because they have a crazy awesome lifetime warranty THAT INCLUDES BATTERIES! Down the road you might also want to get a lightweight 12V drill; they're half the weight and easier to use for small tasks.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/impact-driver.png" alt=""></td>
<td><strong>Impact Driver</strong></td>
<td>Impact drivers are essential for fastening larger items like lag bolts and big screws. Plus they're faster and easier on your wrist.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/screwdriver-bits.png" alt=""></td>
<td><strong>Screwdriver Bits</strong></td>
<td>You don't need a ton, but an assortment is nice.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/drill-bits.png" alt=""></td>
<td><strong>Drill Bits</strong></td>
<td>Again, a small assortment is a good place to start. Make sure to include some spade bits in the kit for making larger holes when you need.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/vise-grip-pliers.png" alt=""></td>
<td><strong>Vise-Grip Locking Pliers</strong></td>
<td>These usually come in assorted sizes, for example a set of 3. They're great when you can't use an exact fit wrench or socket, and won't strip the nut like an adjustable wrench can.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/ratchet-wrench.png" alt=""></td>
<td><strong>Ratchet Socket Wrench Set</strong></td>
<td>I don't actually have these, but I want them. I have a bunch of standard wrenches and sockets, so having a kit like this could essentially replace two sets of tools with one. You might eventually need a traditional socket set too, in both US and Metric sizes.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/pliers.png" alt=""></td>
<td><strong>Plier Set</strong></td>
<td>You want all of these pliers. This kit is the Channellock HD-1 kit, which includes Tongue & Groove pliers, Long-nose pliers, Slip Join pliers, and Cutting pliers.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/level.png" alt=""></td>
<td><strong>Level</strong></td>
<td>I recommend a short (12" or so) and a long level (4-6 feet). The short one is good for small stuff like hanging pictures, while the the long is good for basic construction tasks.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/utility-knife.png" alt=""></td>
<td><strong>Utility Knife</strong></td>
<td>In case you didn't already know, most of them can carry a few replacement blades inside the case.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/step-ladder.png" alt=""></td>
<td><strong>Step Ladder</strong></td>
<td>A lightweight but strong step ladder is a good starting point. I actually don't own one and use an adjustable Gorilla ladder, but it's heavy and cumbersome for typical tasks. It allowed me to use a single ladder for all purposes, although I eventually bought an extension ladder as well. Be careful please.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/hammer.png" alt=""></td>
<td><strong>Hammer</strong></td>
<td>A basic hammer for the basic tasks.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/stud-finder.png" alt=""></td>
<td><strong>Stud Finder</strong></td>
<td>I like the stud finders that use multiple LEDs to show you where the stud is, versus ones that you have to drag to find each edge of the stud. The one pictured here is the Franklin T-6. One tip; make sure the battery is good, because you can get some bad readings with a weak battery.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/pry-bar.png" alt=""></td>
<td><strong>Small Pry Bar</strong></td>
<td>These are handy handy for pulling nails and prying trim, but they're also handy when you need to keep a board or door lifted off the floor with your foot while you do something else with your hands.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/safety-glasses.png" alt=""></td>
<td><strong>Safety Glasses</strong></td>
<td>Please use them when you should. They used to be awful; today they're much better. I bought these 3M Virtua CCS glasses after I had dust in my eye for 2 days while branch trimming despite wearing glasses.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/hearing-protection.png" alt=""></td>
<td><strong>Hearing Protection</strong></td>
<td>I have the big ear muff things too, but these are easier to use. Or you could be an idiot and not notice the hearing loss that occurs when you run a saw over a weekend project.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/tool-belt.png" alt=""></td>
<td><strong>Tool Belt</strong></td>
<td>The more basic and non-intrusive the better in my opinion. They're also kinda hot.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/headlamp.png" alt=""></td>
<td><strong>Headlamp</strong></td>
<td>You probably already have one, but if you don't, get one. You actually don't need top of the line for these, since they're all really bright these days. They beat flashlights because the light is always pointed where you need it, they can illuminate small spaces, and they free both hands.</td>
</tr>
</tbody>
</table>
<h2 id="general-repair-and-construction" tabindex="-1">General Repair and Construction <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/caulk-gun.png" alt=""></td>
<td><strong>Caulk Gun</strong></td>
<td>Once again, get a good one; it costs $5 more than a cheap one. You'll use it for decades.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/mason-chisel.png" alt=""></td>
<td><strong>Mason's Chisel</strong></td>
<td>I say masonry, and it is good for cutting that, but it's also good for smashing against anything you don't want to use a finer tool for. WEAR SAFETY GLASSES!</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/hacksaw.png" alt=""></td>
<td><strong>Hacksaw</strong></td>
<td>Good for cutting plastic and metal pipes.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/staple-gun.png" alt=""></td>
<td><strong>Staple Gun</strong></td>
<td>Useful for stuff like screen repair. I like the ones made of aluminum because they're much lighter than steel.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/air-compressor.png" alt=""></td>
<td><strong>Small Air Compressor</strong></td>
<td>These aren't required, but are handy for filling things up with air and running small air-driven tools. I used to have a giant one but got rid of it because its space-to-usefulness ratio was bad.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/shop-vac.png" alt=""></td>
<td><strong>Shop Vac</strong></td>
<td>I have a big one for project cleanup, and a cheap little one for cleaning out the car.</td>
</tr>
</tbody>
</table>
<h2 id="basic-woodworking" tabindex="-1">Basic Woodworking <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<table>
<thead>
<tr>
<th> </th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/rubber-mallet.png" alt=""></td>
<td><strong>Rubber Mallet</strong></td>
<td>Rubber mallets are for pounding on wood and metal without leaving a mark.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/pull-saw.png" alt=""></td>
<td><strong>Pull Saw</strong></td>
<td>Also called a Japanse Saw, these cut in the pull direction, as opposed to the push direction of a standard hand saw. I like them because I think they're more precise, easier to use, and are more versatile.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/clamps.png" alt=""></td>
<td><strong>Clamps</strong></td>
<td>Clamps firmly hold boards for glueing and cutting. You can never have enough.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/combination-square.png" alt=""></td>
<td><strong>Combination Square</strong></td>
<td>Used to align boards and mark boards for cutting. Used when you can't find your speed square.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/speed-square.png" alt=""></td>
<td><strong>Speed Square</strong></td>
<td>Used to measure angles and mark boards for cutting. Used when you can't find your combination square.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/miter-saw.png" alt=""></td>
<td><strong>Miter Saw and Stand</strong></td>
<td>A miter saw will let you cut boards at an angle. A compound miter saw will let you cut boards with both angle and bevel, which is needed for things like trim mouldings. A sliding saw will let you cut wider boards. I have a sliding compound miter saw and have needed it. I think everyone doing basic wood cuts will want a miter saw, but whether you want a sliding and/or compound miter saw is up to you. A portable stand is great for quick storage and access.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/circular-saw.png" alt=""></td>
<td><strong>Circular Saw</strong></td>
<td>This is for long straight cuts in large boards like plywood.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/jig-saw.png" alt="tool"></td>
<td><strong>Jig Saw</strong></td>
<td>When you need to make hand-guided angle cuts. You can use guides to help you make straight cuts with jig saws as well.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/dremel.png" alt=""></td>
<td><strong>Dremel</strong></td>
<td>The swiss army knife for small cutting and sanding jobs. For the love of Bob Ross please use your safety glasses because these things spin at 35,000 RPMs, and the cutting discs can and do break. I really like the cordless models.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/block-plane.png" alt=""></td>
<td><strong>Block Plane</strong></td>
<td>Another commonly used woodworking tool for shaving and trimming wood.</td>
</tr>
</tbody>
</table>
<h2 id="plumbing" tabindex="-1">Plumbing <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/pipe-cutter.png" alt=""></td>
<td><strong>Pipe Cutter</strong></td>
<td>You use these to cut copper pipes, like the one that you need to replace because they used a globe valve instead of a 1/4 turn valve on it.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/pipe-wrench.png" alt=""></td>
<td><strong>Pipe Wrench</strong></td>
<td>These are designed to bite harder into the pipe as you turn them, so are essential for working with water and gas pipes.</td>
</tr>
</tbody>
</table>
<h2 id="electrical" tabindex="-1">Electrical <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<table>
<thead>
<tr>
<th> </th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/outlet-tester.png" alt=""></td>
<td><strong>Outlet Tester</strong></td>
<td>These tell you if an outlet is wired correctly. Things can get a little confusing, and former homeowners can make some really dumb decisions that don't make sense. Plus sometimes I just screw up, and these things help keep you passing inspection.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/voltage-tester.png" alt=""></td>
<td><strong>Voltage Tester</strong></td>
<td>There are more complex tools, but at the very basic these will tell you if wiring is hot or not. It can be a lifesaver.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/wire-stripper.png" alt=""></td>
<td><strong>Wire Stripper</strong></td>
<td>I originally recommended the Klein 11045 10-18 AWG solid wire stripper for this. I've found it to be better than other wire strippers I've used. But after a friend's recommendation I learned there are more advanced and ergonomic tools like the Klein 11063W that will grip and strip the wire to the correct length in a single action.</td>
</tr>
</tbody>
</table>
<h2 id="drywall" tabindex="-1">Drywall <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/drywall-saw.png" alt=""></td>
<td><strong>Drywall Saw</strong></td>
<td>If you have drywall, this is a good basic tool for making cuts.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/taping-knife.png" alt=""></td>
<td><strong>Taping Knife</strong></td>
<td>When it comes to drywall repairs, the bigger the blade, the easier it is to make a seamless repair with joint compound (AKA drywall mud). So if there's enough room, I like to use these. You'll also want to have a quality stainless steel 6" Joint Knife for smaller areas. Don't use plastic.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/mud-pan.png" alt=""></td>
<td><strong>Mud Pan</strong></td>
<td>You'll need a mud pan long enough to fit the taping knife.</td>
</tr>
</tbody>
</table>
<h2 id="painting" tabindex="-1">Painting <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/roller.png" alt=""></td>
<td><strong>9" Roller</strong></td>
<td>Standard size roller. Use high quality roller covers. I toss my roller covers after a project, although pros will clean and reuse them.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/big-roller.png" alt=""></td>
<td><strong>18" Roller</strong></td>
<td>If you have a big room, or more than one room to paint, you want a big roller. They're double the size of a regular roller and cut the painting time nearly in half. Not that many people know how awesome these things are!</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/brushes.png" alt=""></td>
<td><strong>Brushes</strong></td>
<td>Again, use high quality brushes because you'll have better results and they'll last longer. Clean these using a brush cleaning tool immediately when you're done. There are different sizes and shapes depending on the type of job you're doing, so if you're not sure ask for help at the paint desk.</td>
</tr>
</tbody>
</table>
<h2 id="storage" tabindex="-1">Storage <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<p>When you first start out, you don't need to think too much about storage because you don't have that much stuff. It's helpful to be organized from the start, and grow your storage solutions as you accumulate more.</p>
<table>
<thead>
<tr>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/shelving.png" alt=""></td>
<td><strong>Shelving</strong></td>
<td>I started with using cheap plastic shelving, and it was ok. What I'm currently using is Gladiator shelving from Home Depot. I like it because it's strong, deep, and easy to assemble. I had also bought some cheaper shelving from Northern Tool at the time, and there's a pretty big difference in quality and durability. Get the deepest shelving you can fit.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/wall-mount-racks.png" alt=""></td>
<td><strong>Wall Mount Racks</strong></td>
<td>You'll immediately have a bunch of shovels and rakes and brooms, so these work well to keep them organized and off the floor.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/tool-chest.png" alt=""></td>
<td><strong>Tool Chest</strong></td>
<td>Again you can start out small and grow into bigger solutions if you need. These things fill up quickly though, so I'd suggest getting something a little bigger than you might think you'll need if you have the room.</td>
</tr>
<tr>
<td><img src="https://dantanner.com/homeowner-tools/tool-bag.png" alt=""></td>
<td><strong>Tool Bag</strong></td>
<td>I use a tool holder accessory that goes over a 5-gallon bucket to hold my most-often used tools. I'd rather have a nice little tool bag though.</td>
</tr>
</tbody>
</table>
<h2 id="a-note-on-corded-vs-electric-and-gas" tabindex="-1">A Note On Corded vs. Electric and Gas <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<p>Cordless electric tools have come a long way, and they're my first choice if I don't absolutely need a corded or gas-powered tool. Leaf blower, weed trimmer, lawn mower, even snowblower; they're all electric now and so much better in terms of maintenance and ease of use.</p>
<h2 id="where-to-buy" tabindex="-1">Where To Buy <a class="header-anchor" href="https://dantanner.com/post/essential-homeowner-tools/">#</a></h2>
<p>Buy at your local small shop if you can. Not only because it's supporting a smaller business, but because they can often help you better. Don't be surprised if they have the thing you need that you couldn't find at the big box store too, especially if you need to get creative with a solution.</p>
<p>Hope it helps! Once again, please protect your fingers and eyes, read the manual, watch a video from someone that looks like they know what they're doing, and ask for help when you need it.</p>
Chaperone2020-04-12T00:00:00Zhttps://dantanner.com/post/chaperone/<p>I built a basic but extensible monitoring tool to help people watch their systems. It's called <em>Chaperone</em>, and it's <a href="https://github.com/dtanner/chaperone">open-sourced on github</a>.
It's simple and powerful, letting you execute arbitrary scripts to determine health.</p>
<h2 id="why" tabindex="-1">Why? <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h2>
<p>A couple years ago, a bright colleague of mine whipped up a quick and elegant solution to watch and alert on a bunch of things running in our apps.
We didn't want a big overblown solution, but it also needed to be customizable. I helped out a little.
When the next suite of projects rolled around, I needed something similar. I could've reused the tool we had, but here's why I wanted to create something new:</p>
<ul>
<li>Almost every project needs something that can make basic monitoring easy, and custom monitoring possible. I was willing to make an investment of my own time to make something I loved for the long-term.</li>
<li>Just about everyone these days is writing apps that are deployed as containers. Existing monitoring tools available were not built for this style of deployment, and didn't meet my goals. They're all built as these big centralized and costly to operate long-term installations, which doesn't fit in a decentralized app model where each team is reponsible for building and administering their own monitoring.</li>
<li>The original tool was written in a language that isn't my favorite; this one's written in Kotlin, and my fondness for it encourages me to write more features with it.</li>
</ul>
<h2 id="what-does-it-do" tabindex="-1">What does it do? <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h2>
<p>In a nutshell, you define a bunch of checks, and when the app starts up, it runs them periodically and sends the results to various places you configure.</p>
<p>More specifically, it runs every check you define as a bash command, and if the exit code of the command is zero, things are good. If it's non-zero, things are not good.</p>
<h2 id="isn-t-that-what-some-other-tool-does" tabindex="-1">Isn't that what $SOME_OTHER_TOOL does? <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h2>
<p>Yeahhhh...some tools do precisely that, some have a ton more features, some provide a giant user interface, and some cost a bunch of money.
But whenever I looked for a tool, they all suffered from one of these flaws:</p>
<ul>
<li>too opinionated in how they worked</li>
<li>a giant PITA to configure or run</li>
<li>couldn't be configured to live alongside my project's source code</li>
<li>too hard or confusing to configure and run as a container</li>
</ul>
<h2 id="what-are-the-highlights-of-its-features" tabindex="-1">What are the highlights of its features? <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h2>
<h3 id="defining-checks" tabindex="-1">Defining checks <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h3>
<p>See the docs for the full list of features with examples, but basically you configure a <code>check</code> as a TOML config file and put that in a directory. If you're not familiar with <a href="https://github.com/toml-lang/toml">TOML</a>, it's a simple and obvious configuration file format. Here's the most basic example:</p>
<pre><code class="language-toml">name = "basic example"
command = "ls"
interval = "1m"
timeout = "5s"
</code></pre>
<p>This example runs <code>ls</code> every minute. A more realistic example would be figurately stated as "curl some url and see if it returns a 200", which can easily be written as a one-liner. You don't need an HTTP-specific application for that.</p>
<p>A more advanced example is a templated check, where you run a "template" command that builds a list of dynamic checks, like "get a list of our running apps in the cluster", then "hit each app's health check url". e.g. Here's a real example we use in our project:</p>
<pre><code class="language-toml">description = "app instance health"
# template will output $app-name $env $instance-id $health-check-url
template = "../scripts/list-instances.sh"
name = "$2 - $3" # the name of each check will be $app-name - $env. e.g. foo - dev
command = "../scripts/check-instance.sh" # call this command for each result from the template
interval = "1m"
timeout = "1m"
tags = {app="$1", env="$2", cat="app"}
</code></pre>
<p>The above example uses the concept of bash positional arguments as variables passed from the template to the command. If we had 10 instances running, it would dynamically execute 10 checks. It also uses tags to help organize our health check results.</p>
<p>Another real world template use case is "from our list of kafka topics, check if we can consume from each topic".</p>
<h3 id="outputting-the-results" tabindex="-1">Outputting the results <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h3>
<p>Result output capabilities is a pluggable model. There's a result OutputWriter interface with a single function <code>fun write(checkResult: CheckResult)</code>, and so far two output destinations have been implemented:</p>
<ul>
<li>stdout - sends the results to stdout in a human-readable format</li>
<li>InfluxDB - sends the results to InfluxDB. I'm currently using influx + grafana for our dashboarding and alerting, and it's been a great way to use tools we're already relying heavily on. For example, here's a sample grafana dashboard showing the results of some checks. You can find it and a sample grafana dashboard export in the <a href="https://github.com/dtanner/chaperone/tree/master/example-usage">example-usage</a> section of the project.
<img src="https://dantanner.com/chaperone/sample-check-dashboard.png" alt=""></li>
</ul>
<p>You choose the output options via a global config file that looks like this:</p>
<pre><code class="language-toml">[outputs.stdout]
[outputs.influxdb]
db="metrics"
defaultTags={app="myapp-chaperone"}
uri="http://localhost:8086"
</code></pre>
<p>Some other ideas that haven't been built yet but would be pretty easy are destinations like:</p>
<ul>
<li>CSV file</li>
<li>Common repositories like relational databases, other time series stores, etc...</li>
<li>Custom destination - e.g. a script you've defined that accepts the results and does whatever with it, like put it in some crazy proprietary database you have.</li>
</ul>
<h2 id="how-is-it-run" tabindex="-1">How is it run? <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h2>
<p>Chaperone itself is built and published as a docker image on <a href="https://hub.docker.com/r/edgescope/chaperone">docker hub</a>.
That's really just the shell though - the expectation is that you create your own docker container based on it, adding in your checks and then deploying that to your environment(s). Documentation on exactly how with a sample docker-compose file is included in the <a href="https://github.com/dtanner/chaperone/tree/master/example-usage">example-usage</a>.
Our team's naming convention for each checks source repository is $projectname-chaperone. Each chaperone definition project lives within the org it's watching, so it's version-controlled and happily living alongside the other org's applications.</p>
<p>Again, see the docs to try it out yourself. I made it with the intention that it's really easy to get going and use.
I also tried to make it easy to debug your checks as you build them, knowing that sometimes validation can get complicated. You can add a <code>debug = true</code> option to a check and it'll log out the commands as they're executed via the <code>bash -x</code> flag.
Feedback and ideas are greatly appreciated. As of this writing, there's a couple minor bugs and a few enhancement ideas in the backlog, but we've been using it in production for a while without major issue so far.
I plan to keep extending and investing in this project as long as it's useful to myself and others.</p>
<h2 id="thanks" tabindex="-1">Thanks <a class="header-anchor" href="https://dantanner.com/post/chaperone/">#</a></h2>
<p>Thanks to Nathan Hartwell for his original ideas on the app. Even though I wrote it from scratch in a different language, I started with a lot of his ideas.
Thanks to Ted Naleid for a great idea on how to make templated checks slicker.
I also discovered some really nice utility libraries in this project; thanks to everyone that has contributed to them:</p>
<ul>
<li><a href="https://ajalt.github.io/clikt/">CLIKT</a> - kotlin command line interface handling</li>
<li><a href="https://github.com/uchuhimo/konf">Konf</a> - This made parsing YAML configs super easy. It's a big and powerful project, and normally for smaller apps I'll use my own <a href="https://github.com/dtanner/env-override">env-override</a>, but Konf is really powerful and elegant.</li>
<li><a href="https://github.com/zeroturnaround/zt-exec">zt-exec</a> - Command execution library. I started with implementing my own usage of ProcessBuilder, but there's a ton of ways to cut yourself. This has been both simple and sturdy.</li>
</ul>
Please Don't Teach C to Beginning Programmers2019-05-05T00:00:00Zhttps://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/<h2 id="the-student" tabindex="-1">The Student <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>I have a niece in her first year at a local university. She's never done any programming before, and thinking of becoming an actuary.
That means she should learn at least a little bit of programming in a tool like <code>R</code> or <code>Python</code>, right?
Those seem like good high-level languages in 2019 that are also applicable in that field.</p>
<h2 id="the-phone-call" tabindex="-1">The Phone Call <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>A few weeks ago I got a call from her Dad asking for some help with homework on an introductory (100-level) programming course titled "Problem Solving in the Natural Sciences (using C and Matlab)".
They were on the <code>C</code> portion of the class. My reaction was, <em>some schools still teach C as an introduction to programming?</em>
My niece was currently stuck on the portion of the program where you read a number from the command line.
It had also been 20 years since I wrote any C myself.</p>
<p>Her: <em>When I read a number and print it, the code prints out a different number.</em><br>
Me: <em>What function are you calling to get the number?</em><br>
Her: <em>get char</em><br>
Me: <em>Oh, you're reading in an ascii character.</em><br>
Her: <em>What's an ascii character?</em><br>
Me: <em>Uhhh, don't worry about that for now. (Starts looking up how to read a decimal from the command line.)</em></p>
<h2 id="we-want-more-programmers" tabindex="-1">We Want More Programmers <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>There are many fields currently filled with technology-averse people, when those roles would be much better performed by people with at least conversational coding abilities.
Think about jobs where people do a lot of data analysis using Excel. Many of them could really benefit from some coding skills, even if it's just some scripting.
The book <a href="https://www.amazon.com/Data-Smart-Science-Transform-Information-ebook/dp/B00F0WRXI0">Data Smart</a> was pretty eye-opening for me in terms of power and simplicity.
It shows some really neat ways to discover information about data, and optimize business problems in ways I didn't know were possible.
And these tools are usable with dozens of hours of training, not thousands.</p>
<h2 id="why-is-c-such-a-bad-starter-language" tabindex="-1">Why is C such a bad starter language? <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>Before I begin bashing C in this context, let me clarify: <strong>C is a great language for its correct purpose.</strong>
When you need to build a program that is fast and very efficient with resources, C is still a good language candidate.
All computer science students should learn C (or maybe Rust now?) when learning closer to the metal.</p>
<p><strong>But new students should learn the basics using a friendlier language.</strong>
C is too low-level and has too many sharp edges.
Many university students might just have one or two programming courses total.
Teaching that group of students C will leave many of them thinking all programming sucks, and they won't ever want to be near a compiler again.</p>
<h2 id="let-s-show-some-examples" tabindex="-1">Let's Show Some Examples <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>After helping my niece work through her homework lab, I implemented the assignment myself in three languages: <code>C</code>, <code>Python</code>, and <code>Kotlin</code>.
Python because I'm guessing it's the most common initial language taught in schools, and I think it's a really good teaching language.
Kotlin because I wanted a typed language in the comparison, and I know Java's a popular language taught in schools.
I didn't choose Java because no one ever wants to touch Java after they've used Kotlin.</p>
<h2 id="the-assignment" tabindex="-1">The Assignment <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>The assignment is to make a little program that reads input from the user representing cards in a deck.
The user enters a number to represent each card/suit, then when the user enters <code>-1</code>, the program displays the hand and calculates a score
based on arbitrary scoring rules meant to exercise basic logic and programming.
Implementations here:
<a href="https://gist.github.com/dtanner/b3cf09c59b0fd7e1389f635d817facd6">C</a> |
<a href="https://gist.github.com/dtanner/a7a154a12600519cb1fda47723c3e1f1">Python</a> |
<a href="https://gist.github.com/dtanner/54a8dc0e111d9b62d44fff6501266d1c">Kotlin</a></p>
<h2 id="observations" tabindex="-1">Observations <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<h3 id="c-observations" tabindex="-1">C Observations <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h3>
<ul>
<li><strong>Pointers and Addresses are an unhelpful concept for new programmers</strong>. It's not a trivial concept for most beginners, and obstructs learning the basics of logic.
I think this <a href="https://stackoverflow.com/questions/4025768/what-do-people-find-difficult-about-c-pointers">stackoverflow post</a> illustrates the difficulties well.</li>
<li><strong>Line of code count</strong>. The C implementation is about 60% more lines (143) than either the Python (92) or Kotlin (85) implementations.
Some of it is unnecessary cruft to learners, like function headers. Some of it is due to a smaller language feature set like no collection functions.
A smaller feature set is easier to learn, but I love collection functions because they help make the code more readable and concise.</li>
<li><strong>Testing is an afterthought</strong>. Writing simple automated tests in Python and Kotlin is trivial. e.g. This is a passing test using tooling built into my IDE:</li>
</ul>
<pre><code class="language-kotlin">@Test
fun calculateHandValue() {
assertEquals(22, calculateHandValue(listOf(22, 8, 16, 45, 2)))
assertEquals(1232, calculateHandValue(listOf(47,9,40,48)))
assertEquals(15, calculateHandValue(listOf(24)))
}
</code></pre>
<p>C's test support looks like <a href="https://stackoverflow.com/questions/65820/unit-testing-c-code">a mess</a>.
Compilation hassles (the first example has you running a java program to generate the header file), no built-in support with
a confusing number of choices, and platform incompatibilities will ward off beginners.</p>
<ul>
<li><strong>Language constraints dilute quality of code</strong>.
For example, passing around the array of cards along with its size as a separate parameter in the C implementation makes it less readable and maintainable.
It's practically required though.</li>
</ul>
<pre><code class="language-c">void displayCards(int arr[], int size) {
</code></pre>
<p>Compare this to the same functions in Python and Kotlin:</p>
<pre><code class="language-python">def display_cards(cards):
</code></pre>
<pre><code class="language-kotlin">fun displayCards(cards: List<Int>) {
</code></pre>
<ul>
<li><strong>No strings</strong>. It had been so long I had forgotten C doesn't have strings.
<code>strncpy(suit, "Clubs", 10);</code> ugh. i'm probably not even doing that right, much less worrying about buffer overflow risks.</li>
</ul>
<h3 id="python-observations" tabindex="-1">Python observations <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h3>
<p>It's pretty dang intuitive IMO, even for first-time programmers.
You won't accidentally use <code>getchar()</code> and be left scratching your head.
One thing I missed was static typing. I'm not fluent in Python, and it took me twice as long to write the exercise in Python as it did in Kotlin.
A lot of the time was spent correcting code in the try-it-now stage rather than being highlighted by the editor during the typing stage.</p>
<h3 id="kotlin-observations" tabindex="-1">Kotlin Observations <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h3>
<p>Before writing the Kotlin implementation, I expected it to be more verbose and less readable than Python, because Java.
But IMO it's actually cleaner in some ways.</p>
<ul>
<li><strong>Lambdas in Kotlin are a little cleaner than Python</strong>. And even though they're a higher-level concept that doesn't exist in C, they both make for more readable code than the C equivalent.
e.g. compare these two one-liners in Python and Kotin:</li>
</ul>
<pre><code class="language-python">return sum((get_card_value(card) - get_suit_number(card)) for card in cards)
</code></pre>
<pre><code class="language-kotlin">return cards.sumBy { getCardValue(it) - getSuitNumber(it) }
</code></pre>
<p>Kotlin is the newest language of the bunch, so had more time to incorporate design features from the latest languages.
That said, I wasn't taught lambdas when I was first learning, so don't know if this would be a handy feature or just one more thing to learn for most beginners.</p>
<ul>
<li><strong>Kotlin expands Java's already good Collection support for concise code</strong>. e.g. <code>cards.sumBy</code> and <code>cards.count</code> are one-liners that intuitively implement the function.
In C's syntactic brevity, you write more code and have to manage more state. I speculate this code is harder to read for a beginner compared to the Kotlin equivalent.</li>
</ul>
<pre><code class="language-c">int hasThreeOrMoreNonNumberCards(int arr[], int numCards) {
int count = 0;
for (int i = 0; i < numCards; i++) {
if (getCardValue(arr[i]) >= 11 || getCardValue(arr[i]) == 1) {
count++;
}
}
return count >= 3;
}
</code></pre>
<pre><code class="language-kotlin">fun hasThreeOrMoreNonNumberCards(cards: List<Int>): Boolean {
return cards.count { getCardValue(it) >= 11 || getCardValue(it) == 1 } >= 3
}
</code></pre>
<ul>
<li><strong>Kotlin forces the programmer to think about production scenarios</strong>. This adds a little bit of extra work up front for the programmer,
but saves time in the end with less debugging and errors.
Explicit null awareness and immutability by default are enormous improvements. My C implementation is littered with vulnerabilities and bugs.
One might argue Python is a better introductory language because you can ignore a lot more when focusing on learning and not making a production app. I think this is true.
That said, Kotlin does make it easy to ignore yet still be obvious when you're working with a nullable variables.
The <code>!!</code> in <code>readLine()!!.toInt()</code> indicates we know the input could be null, but we're gonna assume it's not.</li>
<li><strong>Kotlin (a JVM language) requires ceremony</strong>: Without an IDE and Gradle/Maven, things like dependency management and classpaths are burdensome to a beginner.
They're excellent for production use, and IDEs make this a smaller issue, but Python has the least baggage of the bunch.</li>
</ul>
<h2 id="obstacles-and-comparison" tabindex="-1">Obstacles and Comparison <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>I don't know how many schools still teach C to beginners, or what their reasons are.
I asked the chair of the department for my niece's class, and the two main points from the response were:</p>
<ul>
<li>The course is largely a service course for engineering students, as those programs require their students to use C throughout their curriculum.</li>
<li>The course is subject to Engineering program accreditation standards.</li>
</ul>
<p>I obviously disagree with the first point enough to have spent the time to write this article.
I think it's a big deal to start students with the language that will give them the most promise in their future.
To the second reason of accreditation standards, I don't know exactly what this means, but I'd be surprised language choice is a factor in accreditation.
For comparison, here's what some of the top universities in the U.S. teach new students:</p>
<table>
<thead>
<tr>
<th style="text-align:left">School</th>
<th style="text-align:left">Language</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">MIT</td>
<td style="text-align:left"><a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00-introduction-to-computer-science-and-programming-fall-2008/">Python</a></td>
</tr>
<tr>
<td style="text-align:left">Carnegie Mellon</td>
<td style="text-align:left"><a href="https://www.cs.cmu.edu/~15110/syllabus.html">Python</a></td>
</tr>
<tr>
<td style="text-align:left">Stanford</td>
<td style="text-align:left"><a href="https://web.stanford.edu/class/cs106a/">Java</a></td>
</tr>
<tr>
<td style="text-align:left">Cornell</td>
<td style="text-align:left"><a href="http://www.cs.cornell.edu/courses/cs1110/2019sp/">Python</a></td>
</tr>
<tr>
<td style="text-align:left">University of Washington</td>
<td style="text-align:left"><a href="https://courses.cs.washington.edu/courses/cse142/">Java</a></td>
</tr>
</tbody>
</table>
<p>These are some of our best schools, and they all teach either Python or Java to their new students.</p>
<h2 id="paths" tabindex="-1">Paths <a class="header-anchor" href="https://dantanner.com/post/please-dont-teach-c-to-beginning-programmers/">#</a></h2>
<p>This chart shows some typical career paths in today's schools.
In only one of them do I think knowledge of a low-level language like C is important (embedded systems).</p>
<p><img src="https://dantanner.com/intro-programming/language-track.svg" alt=""></p>
<p>I used to have a much narrower view of what constitutes a programmer.
It was a few flavors of people that programmed things like robots, games, and other complete software applications.
Regardless of the field, they all spent most of their time thinking about the programming problem or writing code.</p>
<p>Today the role of programmer includes a broader set of fields.
For example, the deep learning specialist that knows how to build and optimize a model, but doesn't know how to productionize it.
Or the business analyst that knows how to make data-driven predictions with a domain-specific language.
It's OK if they know nothing about pointers.
We <em>do</em> want them to build a solid foundation of good principles that are applicable to most languages.
We <em>don't</em> want them to be sitting at their desk two or ten years into their career and be asked to write code without proper initiation.</p>
<p>So please teach more people programming,
start them off in a language that will put them on the right path,
and encourage them to keep forging those paths.</p>
Meteorologists Urged To Stop Giving 100%2017-10-15T00:00:00Zhttps://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/<p>I recently finished spending a year collecting and comparing forecasts from four different providers for seven different locations across the US.</p>
<h2 id="background" tabindex="-1">Background <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<h3 id="why-did-i-do-it" tabindex="-1">Why Did I Do It? <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>The petty rage of a first world problem. My phone said there was a 0% chance of rain today, but it rained. More than once.
This made me curious about what the forecast data actually looked like, and how different provider predictions compare.</p>
<h3 id="daily-information-analyzed" tabindex="-1">Daily Information Analyzed <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<ul>
<li>Low temp</li>
<li>High temp</li>
<li>Probability of Precipitation (POP). e.g. 70% chance of rain, or 30% chance of snow</li>
</ul>
<h3 id="the-forecast-providers" tabindex="-1">The Forecast Providers <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>I wanted to collect forecasts from all the providers I could find. My requirements were:</p>
<ul>
<li>Must have a free API tier. This ruled out providers like World Weather Online and Weatherbug.</li>
<li>Must provide a numeric probability of precipitation, not just subjective info like "partly cloudy, chance of rain". This ruled out providers like OpenWeatherMap, Yahoo, and Accuweather.</li>
</ul>
<p>Given those requirements, I ended up with four: Aeris, Dark Sky, NOAA, and Weather Underground.</p>
<h3 id="the-locations" tabindex="-1">The Locations <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>I used a limited set of locations, with the following goals:
<strong>I wanted a broad range of weather forecasting difficulty.</strong>
e.g. On one end of the scale was San Diego, which is known to have some of the most stable and predictable weather.
I also wanted the opposite of that, and according to <a href="https://fivethirtyeight.com/features/which-city-has-the-most-unpredictable-weather/">fivethirtyeight</a>, Houghton Michigan
is the least predictable for precipitation.</p>
<p><strong>I wanted the locations to be as specific as possible.</strong>
Providers will give forecasts for a city, which can be misleading for geographically large cities. The most consistent and comparable location I could think of
was airport. Some providers like Wunderground accept an airport code like <code>MSP</code>. The others didn't support airport code, but do support lat/long, so I used that for the rest.</p>
<p>Given those goals, I chose these seven airports:</p>
<ul>
<li>SAN (San Diego International, California)</li>
<li>JFK (John F. Kennedy International, New York)</li>
<li>ORD (O'Hare International, Illinois)</li>
<li>CMX (Houghton County Memorial, Michigan)</li>
<li>MSP (Minneapolis/St. Paul International, Minnesota)</li>
<li>DEN (Denver International, Colorado)</li>
<li>SFO (San Francisco International, California)</li>
</ul>
<h3 id="the-collection-process" tabindex="-1">The Collection Process <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>I built a client that would use the API of each provider, scheduled to collect for all the locations each morning at the same time.
I collected as many days in advance as the provider would give. They all gave predictions from 0 to 3 days out, so 3 days out is the farthest any of my comparisons go.</p>
<p>The date range of the data is thirteen months worth of data: from August 1 2016 through August 31 2017.
There were almost 70,000 rows of data. There were a relatively small number of errors total that resulted in missing data.
e.g. There were a few times where the client was throttled or got some bad JSON, and a few daily observations were missing data.
So the data's not perfect, but seems good enough to make some aggregate observations.
I was pleasantly surprised by the stability of all the services though.
Also kudos to Digital Ocean for 100% uptime for over a year on a $5/month virtual private server.</p>
<p>If you look at the <a href="https://github.com/dtanner/weatherbane">code</a>, you'll see Dark Sky's data labeled as forecast.io - they changed their name midway through my collection process, but I kept
the old name to be consistent.</p>
<h2 id="don-t-bore-us-get-to-the-chorus" tabindex="-1">Don't Bore Us, Get To The Chorus <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>These were the questions I found most interesting to answer...</p>
<h2 id="you-said-it-wouldn-t-rain-today" tabindex="-1">You Said It Wouldn't Rain Today! <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>This chart shows a count of the number of times providers predicted a 0% probability of precipitation today,
when in fact there was at least 0.1 inches of precipitation. Lower is better.
<img src="https://dantanner.com/weather/incorrect-0pct-pop-today.png" alt=""><br>
This is the most lenient way to view this scenario. NOAA and Weather Underground had the highest count at 8 and 7 respectfully.
Aeris and Dark Sky had it happen one time each. This should <strong>never</strong> happen. It looks much worse if you count <em>any</em> precipitation.</p>
<blockquote>
<p><em>But Dan, A zero percent chance of rain doesn't guarantee dry conditions; it just indicates that, based on available information, rain is highly unlikely.</em></p>
</blockquote>
<p>I think this is bullshit. If you're going to give a 0% chance of rain forecast, there should actually be a 0% chance of rain.
If rain is <em>extremely unlikely</em>, then don't give a 0% chance of rain. Give a 1% chance of rain, or a 5% chance of rain. Never 0%.</p>
<h3 id="where-did-these-errors-happen" tabindex="-1">Where did these errors happen? <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/incorrect-0-pop-by-location.png" alt=""><br>
Turns out O'Hare was even more difficult to predict than Houghton for this date range. San Diego's not even on the board.
This confirms you can achieve a great work/life balance in San Diego as a forecaster.</p>
<h3 id="you-were-sure-it-wouldn-t-rain-three-days-ago" tabindex="-1">You Were Sure It Wouldn't Rain Three Days Ago? <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>If you include predictions from <strong>three or less</strong> days out, you end up with these counts by provider.
Again, this is forecasting a 0% chance of precipitation, when it reality there was at least 1/10th an inch of precip.
<img src="https://dantanner.com/weather/incorrect-0-pop-3-days.png" alt=""><br>
NOAA's data is pretty bad for this span - I'm not sure what's going on. The lesson here is don't rely on NOAA's API data for long-range precipitation predictions.</p>
<h2 id="you-were-sure-it-would-rain" tabindex="-1">You Were Sure It Would Rain! <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>Not nearly as serious a cloud crime since you hopefully brought your rain jacket, but still something that should never happen...
<img src="https://dantanner.com/weather/incorrect-100-pop-same-day.png" alt=""><br>
Above is the count by provider of those that predicted a 100% chance of precipitation today, when in reality there was none.</p>
<h2 id="nailed-it" tabindex="-1">Nailed It! <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>Enough complaining. Let's talk about those providers that totally nailed it for various situations.</p>
<h3 id="perfect-no-precipitation-prediction-today" tabindex="-1">Perfect no-precipitation prediction today <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>Here's a chart showing the count by provider for correctly predicting a 0% chance of precipitation today:
<img src="https://dantanner.com/weather/correct-no-precip-that-day.png" alt=""><br>
Often the sunny skies optimist, NOAA unsurprisingly is correct the most often at predicting no precipitation.</p>
<h3 id="rain-long-shots" tabindex="-1">Rain Long-Shots <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>Here's a chart showing the count by provider for correctly being 100% sure about it raining 3 days out:
<img src="https://dantanner.com/weather/rain-long-shots.png" alt=""><br>
This was interesting to me - Wunderground is 100% sure about precipitation from 3 days out quite a bit, and they weren't ever wrong about that.
(Again using 1/10th of an inch as the threshold of substantial precipitation.) They get the bold award.</p>
<h3 id="nailing-today-s-lows" tabindex="-1">Nailing today's lows <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p>A count by provider correctly predicting today's low temp
<img src="https://dantanner.com/weather/perfect-todays-lows.png" alt=""><br>
Also interesting to me - Dark Sky has almost twice the count compared to the other providers in predicting today's low temperature.</p>
<h3 id="nailing-today-s-lows-by-location" tabindex="-1">Nailing today's lows by location <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/perfect-low-prediction-by-location.png" alt=""><br>
You're almost twice as likely to get a perfect low prediction in San Francisco compared to Denver.</p>
<h3 id="nailing-today-s-highs" tabindex="-1">Nailing today's highs <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/perfect-highs.png" alt=""><br>
The range between providers here is much smaller. Where Dark Sky was the best at predicting today's low, they are the worst of the bunch at predicting today's high.</p>
<h2 id="error-mean-absolute-deviation-mad" tabindex="-1">Error Mean Absolute Deviation (MAD) <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>AKA objectively speaking, how far off were the predictions from the actual measurements for the entire date range?
The formula is Sum(Absolue Value of(prediction - actual)) / Number of Predictions.
A zero value would be a perfect prediction every time. A lower value is more accurate.</p>
<h3 id="today-s-low-temp-mad-by-provider" tabindex="-1">Today's low temp MAD by provider <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/mad-lows.png" alt=""><br>
This chart correlates closely with the counts of getting the low temp exactly right, which implies Dark Sky is consistently better at predicting low temps,
with the other providers doing about the same as each other for that category.</p>
<h3 id="today-s-low-temp-mad-by-location" tabindex="-1">Today's low temp MAD by location <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/mad-lows-by-location.png" alt=""><br>
This chart shows Denver as being the most difficult location among the gang to predict for low temps, with San Diego and SFO the easiest.</p>
<h3 id="today-s-high-temp-mad-by-provider" tabindex="-1">Today's high temp MAD by provider <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/mad-highs.png" alt=""><br>
Wunderground on average is the most accurate at getting highs on average. What's mildly interesting is that NOAA and Wunderground tied
on their counts of <em>exactly</em> predicting today's high temp, so Wunderground having a lower MAD means they were on average closer to the actual high.</p>
<h2 id="measuring-precipitation-accuracy" tabindex="-1">Measuring Precipitation Accuracy <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>You can use the MAD calculation for <strong>temperature</strong> predictions to get an objective average score, but it's not as clear how to objectively measure for <strong>precipitation</strong> predictions, since they're probabilities and not absolute values.
I think we can get close though. For example, if you say there's a 30% chance of rain, I say there's a 90% chance of rain, and it doesn't rain, you obviously did a better job of predicting rain than I did.</p>
<p>For cases where it didn't rain, the score for a prediction could be the probability value itself.
e.g. Give you an error score of 0.3 for that prediction, and me a score of 0.9. A lower score indicates a more accurate prediction.</p>
<p>For cases where it did actually rain, the error score would be <code>1 - POP</code>.
e.g. It rained, I predicted 30%, so my error score is <code>1 - 0.3</code>, or <code>0.7</code>.
Using this technique, here's a MAD-like score for today's precipitation prediction by provider for the time frame:
<img src="https://dantanner.com/weather/mad-pop.png" alt="">\</p>
<h2 id="measuring-temperature-accuracy-by-tier" tabindex="-1">Measuring Temperature Accuracy By Tier <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>How about some counts showing the <em>subjective quality</em> of each forecast?
i.e. Let's say that exactly predicting today's temperature is considered <em>perfect</em>, within 1-3 degrees is <em>good</em>, 4-6 is <em>ok</em>, and > 6 is <em>bad</em>.</p>
<h3 id="quality-count-for-today-s-low-temps" tabindex="-1">Quality count for today's low temps <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/low-tiered.png" alt=""><br>
Once again, this chart shows that Dark Sky is considerably better than anyone else at predicting the low temp.</p>
<h3 id="quality-count-for-today-s-high-temps" tabindex="-1">Quality count for today's high temps <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/high-tiered.png" alt=""><br>
The accuracy comparison of the providers is much closer for measuring today's high temp.</p>
<h2 id="distance-from-noaa" tabindex="-1">Distance from NOAA <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>My understanding is that NOAA is our government agency that does the heavy lifting for all the providers,
and then they augment that information with their own analysis to further improve the quality of information.
One question I had was, how much do the providers' forecasts differ from NOAA's?</p>
<h3 id="distance-from-noaa-for-today-s-low" tabindex="-1">Distance from NOAA for today's low <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/noaa-distance-low.png" alt=""><br>
Dark Sky deviates from NOAA over twice as much as the next provider.
This was interesting, given how much better they were at predicting today's low temp compared to everyone else.
This made me wonder if there was a correlation between distance from NOAA and accuracy.</p>
<h3 id="distance-from-noaa-for-today-s-high" tabindex="-1">Distance from NOAA for today's high <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h3>
<p><img src="https://dantanner.com/weather/noaa-distance-high.png" alt=""><br>
Again Dark Sky deviates from NOAA by far more than anyone, but their high temp accuracy was the least accurate.
So being different isn't necessarily better in this case.</p>
<h2 id="conclusion" tabindex="-1">Conclusion <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>This turned out to be more charts that I originally intended, but it was interesting to me to view the data from different angles.
Each provider presents the information in a little different way, with some more accurate in different areas than others.</p>
<p>I did make one change to my phone's weather apps. In the past, I only used Wunderground because it has the best user interface for how I want to see the information.
Since analyzing this data, I've also added Dark Sky to my apps simply because they were so good at low temp predictions.
Their UI is novel, but its hyper-local precipitation predictions actually haven't been more useful or accurate for me than glancing at the radar.</p>
<p>From a novice consumer's perspective on forecasting, I would make these changes:</p>
<ul>
<li>Analyze and adjust those 0% and 100% POP error situations. They shouldn't ever happen. Ever.</li>
<li>License or emulate whatever special sauce Dark Sky puts in today's low temp predictions.</li>
<li>Recognize current limits in long-range forecasting and lessen forecasting confidence for 3 or more days out to be between 20-80% or some other squishy number. No more zero and 100% chance of rain unless we're really really sure.</li>
</ul>
<h2 id="did-i-make-any-mistakes" tabindex="-1">Did I make any mistakes? <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<p>Please let me know if I did and I'll update the article. The code and data is referenced below, and social links are in the footer.</p>
<h2 id="references" tabindex="-1">References <a class="header-anchor" href="https://dantanner.com/post/meteorologists-urged-to-stop-giving-100-percent/">#</a></h2>
<ul>
<li><a href="https://github.com/dtanner/weatherbane">Forecast API Client Project</a></li>
<li><a href="https://github.com/dtanner/weatherbane/raw/master/db/prediction-results-queries.sql">Queries used to collect the data</a></li>
<li><a href="https://github.com/dtanner/weatherbane/raw/master/data/prediction-results.csv.zip">Data Collected</a></li>
<li><a href="https://www.ncdc.noaa.gov/cdo-web/search">NOAA Weather History Downloads</a></li>
</ul>
Camp Lotto2017-08-30T00:00:00Zhttps://dantanner.com/post/camp-lotto/<p>There's a great <a href="https://www.ymcamn.org/camps/camp_du_nord">family camp</a> my family has attended over the years.
We spend a week on a beautiful lake with good friends, pawn our children off for a few hours each day, and disconnect from the world.</p>
<p>It started as a tiny little family camp decades ago, and is now so popular there's a <a href="https://www.ymcamn.org/camps/camp_du_nord/family_camp/summer/summer_reservations__lottery">lottery system</a> held each winter to determine who gets in for the summer.</p>
<p>The lottery process has some major flaws though, and gives a sour taste to an otherwise well-executed operation. Specifically:</p>
<h4 id="it-wastes-time" tabindex="-1">It Wastes Time <a class="header-anchor" href="https://dantanner.com/post/camp-lotto/">#</a></h4>
<p>Applicants register for the lottery online with their preferences, choosing preferred cabins, weeks, and a preference of cabin vs. week (i.e. is the week more important than the cabin?).
This is good.</p>
<p>But the actual lottery happens on a Saturday and requires all applicants to be available from 8am-1pm.
The staff calls each family in order, discusses the options again, and the family makes a choice based on what's available.</p>
<p>As an applicant, blocking out half a Saturday to be available by phone for a frantic mini-planning session is a hassle, and if you miss the call you get bumped down the list.
And I can only imagine how chaotic the office must be.
There are 12 weeks available each summer, with around 45 sites.
That's over 500 potential reservations to make over the phone, followed by an apology call to each family that was waiting all morning and didn't make the lottery.</p>
<h4 id="it-s-unfair" tabindex="-1">It's Unfair <a class="header-anchor" href="https://dantanner.com/post/camp-lotto/">#</a></h4>
<p>So you didn't make the lottery this year. Oh well. Looks like we'll have to vacation on our private island <em>again</em>. Wouldn't it be nice if you were put in the front of the line for next year's lottery?
There's nothing in place currently to allow this though, and some people wait years to get in.</p>
<h4 id="it-doesn-t-produce-the-best-outcome" tabindex="-1">It Doesn't Produce The Best Outcome <a class="header-anchor" href="https://dantanner.com/post/camp-lotto/">#</a></h4>
<p>Applicants each make a different set of choices, and those available choices narrow as the lottery progresses.
In the end, the organization is left with a number of unpicked sites and weeks.
What if the first family chosen in the lottery would've been happy with one of those unpicked sites?
With a manual lottery process, you only get one pass.
Wouldn't it be nice if you could optimize the combination of preferred sites and weeks to produce a lottery that accomodated the most campers?</p>
<h2 id="camp-lotto" tabindex="-1">Camp Lotto <a class="header-anchor" href="https://dantanner.com/post/camp-lotto/">#</a></h2>
<p>Thus was born <a href="https://github.com/dtanner/camplotto">Camp Lotto</a> - an attempt to solve these problems.
At a high level, it takes a list of available reservations and lottery registrations, and produces a list of reservations.
The process is similar to a <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo simulation</a>,
in that it simulates the lottery many times. The winner is the lottery simulation that leaves the fewest number of available reservations.</p>
<h4 id="features" tabindex="-1">Features <a class="header-anchor" href="https://dantanner.com/post/camp-lotto/">#</a></h4>
<ul>
<li>Applicants give 3 pieces of info: an ordered list of acceptable sites, an ordered list of acceptable weeks, and a preference indicating if their week or site should be prioritized when finding the next available reservation.</li>
<li>You can mark applicants as having priority (e.g. those people that didn't get in last year). Every time it runs the lottery simulation, it places them at the top of the shuffled list.</li>
<li>You can mark sites as unavailable for certain weeks. e.g. Scheduled repairs or other known unavailabilies.</li>
<li>Along with producing a spreadsheet of reservations, it also produces a sheet of any sites still open and unmatched registrations.
Those remaining sites could be made available for open enrollment, and next year all the unmatched registrations could be given priority.</li>
</ul>
<h4 id="input-and-output-examples" tabindex="-1">Input and Output Examples <a class="header-anchor" href="https://dantanner.com/post/camp-lotto/">#</a></h4>
<p>The first iteration of the app uses a spreadsheet for input and output, because everyone knows how to use a spreadsheet. Here's what the available reservations sheet looks like:
<img src="https://dantanner.com/camplotto/input-available-reservations.png" alt=""><br>
An <code>x</code> in the cell indicates the site is available that week. In the above example, Site B is considered unavailable for Week 2.</p>
<p>Next is the lottery registration information. In reality the preferences will be all over the place, but you get the idea.<br>
<img src="https://dantanner.com/camplotto/input-registrations.png" alt=""></p>
<p>When the program is run, it produces the following spreadsheet. The first sheet consists of the calculated reservations:<br>
<img src="https://dantanner.com/camplotto/output-reservations.png" alt=""></p>
<p>Any unreserved sites would be listed in the <em>Sites Still Open</em> tab, and any unmatched registrations would look like this:<br>
<img src="https://dantanner.com/camplotto/output-unmatched-registrations.png" alt=""></p>
<h2 id="next-steps" tabindex="-1">Next Steps <a class="header-anchor" href="https://dantanner.com/post/camp-lotto/">#</a></h2>
<p>The project is open source (GPL-3 licensed). Currently it's run as a java app on your machine.
One obvious next step is to have it run on a server and let people upload a spreadsheet to it through a web app, or integrate through some other type of data format.</p>
<p>Hopefully my favorite summer camp will use it.
I pinged them about it just before the busy summer season began, and they seemed excited, but we haven't reconnected yet.
So until they adopt this new system, please disregard all of my praise for the camp.
It's actually a terrible mosquito-infested nightmare, completely devoid of wifi, and you shouldn't even think about going there.</p>
Mountain Bike Frame Hydration Hack2017-08-11T00:00:00Zhttps://dantanner.com/post/frame-hydration/<p>I've been carrying water on my frame using a custom solution the last couple seasons.
I originally googled for a solution and didn't find anything commercially available, but found lots of helpful posts in various
forums explaining what folks have done. It's worked really well, so thought I'd share a bundle of details in case anyone else is interested.
If you know of an existing commercial solution that does all this, let me know and I'll update the post.</p>
<h2 id="where-this-works-well" tabindex="-1">Where This Works Well <a class="header-anchor" href="https://dantanner.com/post/frame-hydration/">#</a></h2>
<p>The target audience is people that want to carry up to 70 oz of water on trails, but don't want to use a backpack.
70 oz is perfect for most rides up to a few hours.
Backpacks aren't ideal because they're heavy and sweaty.
Bottles on a rough trail are terrible because it's a great way to crash. A couple bottles can only carry at most 50 oz anyway.
This solution also presumes your frame can fit a pack.
In this example I'm using a hardtail, but many full suspension bikes can use a similar solution.</p>
<h2 id="parts" tabindex="-1">Parts <a class="header-anchor" href="https://dantanner.com/post/frame-hydration/">#</a></h2>
<ul>
<li><a href="http://banjobrothers.com/products/current/frame-packs/frame-pack-medium/">Banjo Brothers Medium Frame Pack</a> - $35. Or use whatever frame pack you like.</li>
<li><a href="https://www.platy.com/catalog/product/view/id/16837/s/hoser/category/45">Platypus Hoser Hydration System - 2 Liter</a> - $25. When full it just barely fits in the medium frame pack.
The key to this bladder is that it has a small cap. A <a href="https://dantanner.com/hydration/camelbak-bladder-annot.jpg">CamelBak bladder</a> won't fit in this pack because of its giant cap.</li>
<li><a href="https://www.amazon.com/Camelbak-Quick-LinkTM-Conversion-Kit/dp/B006IB9TSE">CambelBak Quick Link Conversion Kit</a> - $10. Lets you easily remove the bladder and leave the hose in place.</li>
<li><a href="https://www.amazon.com/Retractable-Carabiner-Keychain-Fishing-Translucent/dp/B01EIQXG66">Retractable Badge Holder</a> - $10 for 4, or free if your employer will give you one. They last about a season.</li>
<li><a href="https://www.rei.com/product/637547/camelbak-big-bite-valve">CamelBak Big Bite Valve</a> - $6. The flow of the <a href="https://dantanner.com/hydration/platypus-bite-valve-annot.jpg">Platypus bite valve</a> that comes with the kit is ridiculously slow. Maybe they've improved it since I bought one, but the CamelBak valve works great.</li>
</ul>
<h2 id="steps" tabindex="-1">Steps <a class="header-anchor" href="https://dantanner.com/post/frame-hydration/">#</a></h2>
<ol>
<li>If you're using the Banjo Brothers frame pack, you'll need to cut a little hole in the front for the hose before installing it.
I also used a lighter to singe/seal the hole edges to prevent fraying.
<img src="https://dantanner.com/hydration/hole.jpg" alt=""></li>
<li>If you're using the quick link, install it so it fits in the frame pack. Here's a picture of where I placed it in the line.
The image also shows the orientation of how the bladder sits in the pack.
<img src="https://dantanner.com/hydration/quick-link.jpg" alt=""></li>
<li>Install the frame pack.</li>
<li>Fasten the badge holder onto the bars so the line is facing you as you're riding. I used gorilla tape. Use whatever works.</li>
<li>Route the tubing, and mount it to the badge holder like this.
<img src="https://dantanner.com/hydration/tube-holder.jpg" alt=""></li>
<li>Profit! Well, actually cost you around $80. But sure beats carrying water on your back or eating dirt.</li>
</ol>
Markdown Notes2017-07-16T00:00:00Zhttps://dantanner.com/post/markdown-notes/<p><strong>tl;dr; markdown files + file syncing + built-in OS file searching = easy organized notes</strong></p>
<p>I've been doing more devops work the last couple years,
which requires a broader and shallow level of knowledge vs. writing code in one or two languages for months at a time.</p>
<p>Because of the dozens of tools I need to juggle, because I forget things, and because I forget things, keeping good notes is essential.</p>
<p>I'm surprised at how many programmers don't keep any notes at all, so hopefully this will convince a couple people to invest a little in their flow.
Thanks to <a href="http://naleid.com/">Ted Naleid</a> for showing me his workflow; it motivated me to properly organize my notes.</p>
<h2 id="history" tabindex="-1">History <a class="header-anchor" href="https://dantanner.com/post/markdown-notes/">#</a></h2>
<p>I've used a few different combination of styles for keeping programming notes over the years:</p>
<ul>
<li>whatever notepads were available in the office supply cabinet</li>
<li>moleskine notebook + space pen during my hipster Getting Things Done phase</li>
<li>.txt files semi-randomly organized on my computer</li>
<li>various notes programs like OneNote, Evernote, and Google Keep</li>
</ul>
<p>I've never been content with any of them. None of them had all these features:</p>
<ul>
<li>easy</li>
<li>fast</li>
<li>copy/pasteable</li>
<li>reliable</li>
<li>flexible</li>
<li>durable</li>
<li>portable</li>
</ul>
<p>Here's what works for me:</p>
<h2 id="step-1-choose-your-file-format" tabindex="-1">Step 1 - Choose Your File Format <a class="header-anchor" href="https://dantanner.com/post/markdown-notes/">#</a></h2>
<p>I use markdown. I would suggest that unless you really like some competing but similarly simple format.
Don't use a binary format.
Don't use HTML.
Don't use a format that will be hard to programmatically convert from ten years from now when it becomes obsolete.</p>
<h2 id="step-2-choose-your-file-syncing-tool" tabindex="-1">Step 2 - Choose Your File Syncing Tool <a class="header-anchor" href="https://dantanner.com/post/markdown-notes/">#</a></h2>
<p>I use dropbox, and keep my notes in <code>/Dropbox/code/notes</code>.</p>
<h2 id="step-3-organization" tabindex="-1">Step 3 - Organization <a class="header-anchor" href="https://dantanner.com/post/markdown-notes/">#</a></h2>
<p>Now it's just a matter of creating your files and filling them up as you go along.
I have a file per tool or concept, and keep project-specific notes in their folder. e.g.:</p>
<pre><code class="language-bash">➜ notes tree
.
├── cassandra.md
├── consul.md
├── curl.md
├── docker.md
├── drone.md
...
├── linux-debugging.md
...
├── weatherbane
├── weatherbane-notes.md
...
</code></pre>
<h2 id="step-4-workflow-notes" tabindex="-1">Step 4 - Workflow Notes <a class="header-anchor" href="https://dantanner.com/post/markdown-notes/">#</a></h2>
<p>I can usually remember the name of the file I want to open, and that's how I usually access the files.
e.g. with Alfred on OS X, where Cmd-Space is the hotkey to bring up the search box:
<em>Cmd-Space</em> <code>open postg</code> and then the tool will autocomplete to postgres.md, and I'll hit enter to open the file in my favorite text editor.</p>
<p>On my Android phone, I use JotterPad to view and (rarely) edit notes.</p>
<h2 id="more-ideas" tabindex="-1">More Ideas <a class="header-anchor" href="https://dantanner.com/post/markdown-notes/">#</a></h2>
<ul>
<li><a href="https://www.reddit.com/r/archlinux/comments/3a0ibj/notetaking_and_markdown/">https://www.reddit.com/r/archlinux/comments/3a0ibj/notetaking_and_markdown/</a></li>
<li><a href="http://lifehacker.com/5943320/what-is-markdown-and-why-is-it-better-for-my-to-do-lists-and-notes">http://lifehacker.com/5943320/what-is-markdown-and-why-is-it-better-for-my-to-do-lists-and-notes</a></li>
</ul>
curl saves the day2017-07-15T00:00:00Zhttps://dantanner.com/post/curl-saves-the-day/<p>Every developer should be comfortable with <a href="https://curl.haxx.se">curl</a>.</p>
<p>For the same reason that you should at least be conversational in tools like sed, awk, and vi, you should be able to quickly type in something like
like <code>curl -v localhost:8080</code> to see what's going on. It's available on just about every unix-based box for a good reason.</p>
<p>Like its man page states, the number of features will make your head spin. That's probably the most discussed reason people are turned off by it.
You don't have to understand all its features though; learn five of them and you'll be able to quickly solve problems you couldn't before.</p>
<p>Memorize the basics, and keep notes of the more advanced problems you solve with it for the future.</p>
<h2 id="the-basics" tabindex="-1">The Basics <a class="header-anchor" href="https://dantanner.com/post/curl-saves-the-day/">#</a></h2>
<p><strong>GET something</strong><br>
<code>curl https://jsonplaceholder.typicode.com/posts/1</code></p>
<p><strong>Verbose output [-v]</strong> (e.g. show response status, headers, etc..)<br>
<code>curl -v https://jsonplaceholder.typicode.com/posts/1</code></p>
<p><strong>Allow insecure SSL connections [-k]</strong> (useful for self-signed certs you want to temporarily ignore)<br>
<code>curl -k https://untrusted-root.badssl.com/</code><br>
<em>(try it without <code>-k</code> to watch it fail)</em></p>
<p><strong>Add a header [-H name: value]</strong><br>
<code>curl -H "Content-Type: application/json" https://jsonplaceholder.typicode.com/posts/1</code></p>
<p><strong>POST a file [-X POST and -d @file-path]</strong><br>
<code>curl -X POST -d @/tmp/foo.json https://jsonplaceholder.typicode.com/posts/1</code></p>
<h2 id="more-advanced-usage" tabindex="-1">More Advanced Usage <a class="header-anchor" href="https://dantanner.com/post/curl-saves-the-day/">#</a></h2>
<p>If you leave with nothing but the knowledge of what's above, you'll be in great shape.
You'll be able to quickly solve situations with minimal effort and tooling. But keep reading to get a glimpse of advanced features.</p>
<p>Below are examples of commands I keep in my /notes/curl.md file.</p>
<h3 id="get-a-service-100-times-and-get-a-count-of-the-response-codes-returned" tabindex="-1">Get a service 100 times, and get a count of the response codes returned <a class="header-anchor" href="https://dantanner.com/post/curl-saves-the-day/">#</a></h3>
<p>This is really handy when you have a service that is misbehaving, and you want to get a quick idea of how it's responding beyond a single request.
For example, you might find that 25% of requests fail, then discover that 1/4 of your servers are misconfigured.</p>
<pre><code class="language-bash">curl -sL -w "%{http_code}\\n" -o /dev/null "https://foo.com/?[1-100]" | sort | uniq -c
23 200
77 503
</code></pre>
<p>Explanation of the arguments:<br>
<code>-s</code>: silent/quiet mode. don't show progress meter or error messages.<br>
<code>-L</code>: follow redirects<br>
<code>-w "%{http_code}\\n"</code>: make curl write out the http code of the response, followed by a new line<br>
<code>-o /dev/null</code>: write the response to a file (in this /dev/null, meaning we don't care)<br>
<code>?[1-100]</code>: [] is a range specifier. In this case, execute a request for every value in the range. e.g. <code>foo.com?1</code>, <code>foo.com?2</code>, etc...</p>
<p>Then we pipe the response code output to <code>sort</code> and <code>uniq -c</code>, which gives us a count of each response code.
From the above sample output, there were 23 responses with a 200 response code, and 77 responses with a 503 response code. Not good!</p>
<p>Also note that the URL in this example is quoted. This enables us to use special characters like the <code>?</code> query parameter without having to escape them.</p>
<p>Similar to the range specifier is the sequence operator, e.g. <code>{a, b}</code>.
e.g. Type one curl command, but have it use a few different argument values, or have it hit each of your servers individually.</p>
<h3 id="get-response-time" tabindex="-1">Get response time <a class="header-anchor" href="https://dantanner.com/post/curl-saves-the-day/">#</a></h3>
<p>Similar to the above example, but just returns the total response time in seconds.</p>
<pre><code class="language-bash">curl -sL -w "%{time_total}\\n" -o /dev/null "https://jsonplaceholder.typicode.com/posts/1"
1.412
</code></pre>
<h3 id="measure-network-performance-for-the-request-segments" tabindex="-1">Measure network performance for the request segments <a class="header-anchor" href="https://dantanner.com/post/curl-saves-the-day/">#</a></h3>
<p>The following example is really useful when want to quickly see where your request is taking its time.</p>
<pre><code class="language-bash">curl -sL -w ' namelookup: %{time_namelookup}\n connect: %{time_connect}\n appconnect: %{time_appconnect}\n pretransfer: %{time_pretransfer}\n redirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\n total: %{time_total}\n\n' -o /dev/null "https://jsonplaceholder.typicode.com/posts/1"
namelookup: 0.071
connect: 0.109
appconnect: 0.349
pretransfer: 0.349
redirect: 0.000
starttransfer: 0.379
total: 0.379
</code></pre>
<h2 id="a-case-of-curl-saving-my-bacon" tabindex="-1">A case of curl saving my bacon <a class="header-anchor" href="https://dantanner.com/post/curl-saves-the-day/">#</a></h2>
<p>Recently our team deployed a new mobile app, to a new company warehouse, using a new network provider, in a new server infrastructure.
The users reported occasional slow response times in the application. ugh - so many variables.
Our server API stats were all consistently fast. Load testing our services showed adequate performance, both from the warehouse and from headquarters.
What also made this difficult was that during the alpha testing, initial actual usage was so low that our initial performance metrics didn't show any patterns.</p>
<p>Given that we couldn't reproduce the issue internally, but could see it was only happening in the warehouse on the warehouse mobile device network, implied a network issue.
But the network support staff couldn't find any issues, so we were stumped for a little while.</p>
<p>I went to the warehouse to help support the rollout and used the above examples to eventually find the problem.
This was the first command that pointed me toward the real issue:</p>
<pre><code class="language-bash">while true; do curl -sL -w "%{time_total}\\n" -o /dev/null 'https://oursite.com/health'; sleep 5; done
0.293
0.298
0.426
0.292
0.833
5.823
0.283
0.296
0.296
0.293
0.295
0.553
0.294
0.817
0.289
0.295
0.300
5.828
0.364
0.300
...
</code></pre>
<p>Notice any pattern? Every minute we got a slow response! That smells like a lookup or a caching issue.
So I issued this command to measure network segment performance:</p>
<pre><code class="language-bash">curl -sL -w ' namelookup: %{time_namelookup}\n connect: %{time_connect}\n appconnect: %{time_appconnect}\n pretransfer: %{time_pretransfer}\n redirect: %{time_redirect}\nstarttransfer: %{time_starttransfer}\n total: %{time_total}\n\n' -o /dev/null "https://oursite.com/health"
namelookup: 5.543
connect: 5.588
appconnect: 5.770
pretransfer: 5.770
redirect: 0.000
starttransfer: 5.823
total: 5.823
</code></pre>
<p><code>namelookup</code> indicates the amount of time it took to do a DNS name lookup. i.e. The primary DNS server configured for the device was failing to find the server.
The backup server <em>was</em> working though. The device cached the IP address of the server for a minute, and when the cache expired, the process repeated itself.
Using a different DNS server while we fixed the underlying DNS issue solved the problem.</p>
<p><em>It was DNS.</em></p>
<h2 id="summary" tabindex="-1">Summary <a class="header-anchor" href="https://dantanner.com/post/curl-saves-the-day/">#</a></h2>
<ul>
<li>curl's basic and most common features are easy to use and memorize</li>
<li>it has a huge feature list</li>
<li>it will almost always be available on the *nix server you're shelled in to</li>
<li>it integrates beautifully with other shell commands to build up your desired solution</li>
</ul>
<p>curl has dozens of arguments for its features, and it's easy to get lost in its man pages, but it's good stuff.
And since curl is available on just about every system I use, it's been worth the small investment of time for me.
Here's the official page for its <a href="https://ec.haxx.se/cmdline-options.html">command line options</a>.</p>
Groovy is still better than Java2017-01-25T00:00:00Zhttps://dantanner.com/post/groovy-is-still-better-than-java/<p>The idea for this post started last week when a gifted teammate, whose experience lies outside JVM languages, asked me if it was still worth it to write Groovy over Java. Good question...</p>
<p>For the last five years, when it comes to writing for the JVM, it's been mostly Groovy for me.
Crap, it's been almost ten years. <em>dan feels his bones creaking...</em></p>
<p>When I first saw Venkat Subramaniam gliding through his beautiful java to groovy idioms at a conference, I was impressed.</p>
<p>Little things were nice, like being able to use <code>println 'hello'</code> instead of <code>System.out.println("hello");</code>.</p>
<p>Even better was reading a file line by line. It went from this:</p>
<pre><code class="language-groovy">BufferedReader br = new BufferedReader(new FileReader("file.txt"));
try {
StringBuilder sb = new StringBuilder();
String line = br.readLine();
while (line != null) {
sb.append(line);
sb.append(System.lineSeparator());
line = br.readLine();
System.out.println("line = " + line);
}
} finally {
if (br != null) {
br.close();
}
}
</code></pre>
<p>to this:</p>
<pre><code class="language-groovy">new File("file.txt").eachLine { println "line = $it" }
</code></pre>
<p><em>Look Ma, a one-liner!</em> That got me hooked.</p>
<p>The <strong>biggest</strong> thing though, which I've grown to appreciate even more over the years, is the <a href="http://docs.groovy-lang.org/latest/html/groovy-jdk/java/util/Collection.html">Collection</a> interface. The intrinsic power it gives to collection-like objects is immense. And hey, c'mon over here <a href="http://groovy-lang.org/groovy-dev-kit.html#_iterating_on_maps">Map</a> - you're iterable too as far as we're concerned.</p>
<p>There's basic stuff like <code>10.times { println it }</code>, and
<code>someCollection.each { println it }</code>, which shaves a few lines of code from the existing java iteration techniques.</p>
<p>Then there's commonly useful but more powerful improvements like <code>collect()</code>, where you iterate through a collection of things, creating a new collection based on a function. e.g. going from this pre-Java 8 code:</p>
<pre><code class="language-groovy">List<Integer> input = Arrays.asList(1, 2, 3);
List results = new ArrayList();
for (Iterator<Integer> iterator = input.iterator(); iterator.hasNext(); ) {
Integer integer = iterator.next();
results.add(integer * 2);
}
</code></pre>
<p>to:</p>
<pre><code class="language-groovy">List results = [1, 2, 3].collect { it * 2 }
</code></pre>
<p>Filtering a collection is a similar improvement, where you start with a collection and filter it down based on a condition applied to each element. e.g. going from (once again, pre-Java 8):</p>
<pre><code class="language-groovy">List<Integer> input = Arrays.asList(1, 2, 3, 4);
List evenNumbers = new ArrayList();
for (Iterator<Integer> iterator = input.iterator(); iterator.hasNext(); ) {
Integer integer = iterator.next();
if (integer % 2 == 0) {
evenNumbers.add(integer);
}
}
</code></pre>
<p>to:</p>
<pre><code class="language-groovy">List evenNumbers = [1, 2, 3, 4].findAll { it % 2 == 0 }
</code></pre>
<p>It gets even more concise and powerful with methods like <code>collectMany</code> and <code>inject</code>. They're slightly more complicated, but worth the couple minutes of time it takes to understand. If I were to give one piece of advice to people new to Groovy, it would be to thoroughly explore that interface.</p>
<p>Lispers may smugly remark, "oh, how cute; you're learning functional programming". And they're right!
The pragmatic functional features of Groovy provide a gentle transition from object-oriented to more pure functional programming for many developers.</p>
<p>Fast-forward a bunch of years, and things have changed. Even though Java hasn't been progressing as fast as many would like, they've made some significant changes to include features introduced by Groovy, and pioneered by other languages; most notably lambdas and streaming. For example, the above Java examples can now be written like this:</p>
<pre><code class="language-groovy">// read each line in a file
try (Stream<String> stream = Files.lines(Paths.get("file.txt"))) {
stream.forEach(System.out::println);
} catch (IOException e) {
e.printStackTrace();
}
// collect
List<Integer> results = Stream.of(1, 2, 3).map(v -> v * 2).collect(Collectors.toList());
// filter
List<Integer> evenNumbers = Stream.of(1, 2, 3, 4).filter(v -> v % 2 == 0).collect(Collectors.toList());
</code></pre>
<p>...<strong>much</strong> improved over the old Java syntax. Still not as concise or elegant as Groovy, but fairly close.</p>
<h3 id="back-to-the-original-question" tabindex="-1">Back to the original question... <a class="header-anchor" href="https://dantanner.com/post/groovy-is-still-better-than-java/">#</a></h3>
<p>Is it still worth it to write groovy? Fortunately, I had a chance to make a direct comparison that same week.</p>
<p>It started with a little bake-off we're doing to compare a couple different application design techniques. I wrote a small service in plain groovy, and my teammate wrote one in java. Our apps are running in a container that will have their environment-specific overrides specified as environment variables. I strongly prefer typed configuration objects in the code, rather than things like generic Maps or JSON objects. Ten minutes of googling, and I couldn't find a utility to let me override typed configuration object values with environment variables. So I created this project in groovy: https://github.com/dtanner/env-config-loader-groovy</p>
<p>My teammate initially used the <a href="https://github.com/typesafehub/config">typesafe/config</a> project to do his configuration overrides. But its environment overrides are a little verbose IMO, we didn't need its additional complexity, and solving the same problem with different tools adds to the maintenance costs for the team.
So I created a pure java version of the env-config-loader tool as well: https://github.com/dtanner/env-override</p>
<p>I wasn't excited about writing the java version, but:</p>
<ul>
<li>A pure java implementation can be used by other JVM languages</li>
<li>It seemed like a useful tool for the community</li>
<li>I'd be able to have a current and relevant opinion on Groovy's usefulness compared to Java</li>
</ul>
<p>An example line in the Groovy code is this:</p>
<pre><code class="language-groovy">Map<String, String> envOverridesMap = getenv().findAll { it.key.startsWith(environmentPrefix) }
</code></pre>
<p>The above line populates a Map from all system environment variables that match a given environment prefix. I wanted to avoid dependencies if possible to limit the transitive dependencies needed by consumers of the library; the only compile dependency is logback-classic.</p>
<p>On the java side, I also tried building the tool without any dependencies, but it got ugly pretty fast. Replicating the above Groovy wasn't too bad; you've basically seen what it looks like in the earlier collect example:</p>
<pre><code class="language-groovy">Map<String, String> envOverridesMap = new HashMap<>();
for (String envKey : envVars.keySet()) {
if (envKey.startsWith(environmentPrefix)) {
envOverridesMap.put(envKey, envVars.get(envKey));
}
}
</code></pre>
<p>Isn't there a more concise way? IntelliJ suggested this alternative syntax:</p>
<pre><code class="language-groovy">Map<String, String> envOverridesMap = new HashMap<>();
envVars.keySet().stream().filter(envKey ->
envKey.startsWith(environmentPrefix)).forEach(envKey ->
envOverridesMap.put(envKey, envVars.get(envKey)));
</code></pre>
<p>...but that just made me sad. I enjoy the new streaming capabilities, but it didn't add value in for me in this case. It could've been much cleaner if Map were iterable in java, but it's not and probably won't ever be.</p>
<p>Things got ugly when I tried cloning an arbitrary object, and also dynamically setting properties on an object. In Java <code>clone()</code> is a protected method, so you can't call it unless you extend from that object. I tried to hack something together like this:</p>
<pre><code class="language-groovy">private static <T> T cloneObject(T obj) {
try {
T clone = (T) obj.getClass().newInstance();
for (Field field : obj.getClass().getDeclaredFields()) {
if (Modifier.isFinal(field.getModifiers())) {
continue;
}
field.setAccessible(true);
field.set(clone, field.get(obj));
}
return clone;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
</code></pre>
<p>but there were exceptional cases I hadn't handled yet, so eventually gave up and added the commons BeanUtils library so I could use the <code>cloneBean()</code> method. Even uglier was going down the path of dynamic property value overrides. I had started to toy with this monstrosity:</p>
<pre><code class="language-groovy">private static boolean setProperty(Object object, String fieldName, String fieldValueString) {
Class<?> clazz = object.getClass();
try {
Field field = clazz.getDeclaredField(fieldName);
field.setAccessible(true);
Class fieldType = field.getType();
if (fieldType.equals(Integer.class) || fieldType.equals(Integer.TYPE)) {
field.setInt(object, Integer.parseInt(fieldValueString));
} else if (fieldType.equals(Long.class) || fieldType.equals(Long.TYPE)) {
field.setLong(object, Long.parseLong(fieldValueString));
} else if (fieldType.equals(Boolean.class) || fieldType.equals(Boolean.TYPE)) {
field.setBoolean(object, Boolean.parseBoolean(fieldValueString));
} else if (fieldType.equals(Long.class) || fieldType.equals(Long.TYPE)) {
field.set(object, Long.parseLong(fieldValueString));
} else {
Class.forName(field.getDeclaringClass().getName());
field.getDeclaringClass().getConstructor(String.class).newInstance(fieldValueString);
field.set(object, field.getType().getConstructor(String.class).newInstance(fieldValueString));
}
return true;
} catch (NoSuchFieldException e) {
log.warn("Environment override for property " + fieldName + " found, but no matching property exists.");
} catch (Exception e) {
throw new IllegalStateException(e);
}
return false;
}
</code></pre>
<p>...before punting. Since I had already accepted the BeanUtils dependency, I was able to use the <code>copyProperty</code> method and delete a lot of potentially brittle code.</p>
<h3 id="summary" tabindex="-1">Summary <a class="header-anchor" href="https://dantanner.com/post/groovy-is-still-better-than-java/">#</a></h3>
<p>You probably enjoyed reading the java code as much as I enjoyed writing it. It's not the semicolons or the verbose exception handling. It was that I could do in a single line of Groovy what took an entire method in the equivalent Java. This is a big deal in terms of productivity, maintenance, and code quality.</p>
<p>There are other advantages to using Groovy, for example listed by Peter Ledbrook <a href="http://blog.cacoethes.co.uk/groovyandgrails/groovy-in-light-of-java-8">here</a>.
As to the question of when it's appropriate to write Java over Groovy, there's really only a couple reasons in my opinion:</p>
<p>The first is the reason I just insulted the java code I wrote, but still intend to promote and deprecate the groovy version; it's a library that can be run on any JVM language. The Groovy library is limited to Groovy usage. It's worth it for that situation.</p>
<p>The second is if you're using a framework or library that doesn't let you easily use Groovy. If you're going down this path, I think you'll know it when you see it. e.g. If you want to write a lot of RxJava, you might want to stick with Java, at least for now.</p>
<p>tl;dr; I still think Groovy is still generally a better Java.</p>
java.time for the extremely impatient2016-04-13T00:00:00Zhttps://dantanner.com/post/javatime-for-impatient/<p>This is a super quick history and primer for those familiar with java, but unfamiliar
with <a href="http://www.joda.org/joda-time/">Joda-Time</a>.
Java’s original date and time handling had some issues that could make it cumbersome and error-prone to work with. Then
came the Joda-Time library, which is a fantastic replacement for pretty much everything date and time related. It’s
intuitive, clean, fully-featured, and performant.</p>
<p>Starting with Java 8, the library was folded into the core JDK, with very few modifications to the Joda API.</p>
<p>In short, if you’re not on Java 8 yet, you should probably be using the joda-time library. If you are on Java 8, you
should use the java.time classes.</p>
<p>There’s much more to the package than what I’ll show, and you’ll eventually want to dive deeper, but this article shows
you some of the most frequently used techniques I’ve experienced on the last few projects. I’m using groovy for the code
snippets, but they’re just java without the semicolons.</p>
<h4 id="91-83-of-the-time-you-ll-work-with-localdate-or-zoneddatetime" tabindex="-1">91.83% of the time, you’ll work with LocalDate or ZonedDateTime. <a class="header-anchor" href="https://dantanner.com/post/javatime-for-impatient/">#</a></h4>
<pre><code class="language-java">import java.time.LocalDate
import java.time.ZonedDateTime
import java.time.ZoneId
import java.time.format.DateTimeFormatter
import java.time.temporal.TemporalAdjusters
// If you have a date without time information, use a LocalDate. e.g. someone's birthday
LocalDate localDate = new LocalDate(2016, 4, 12)
println localDate.toString() // 2016-04-12
// If you need to include time in the date, use ZonedDateTime
ZonedDateTime zdt = ZonedDateTime.now()
// when formatting a ZonedDateTime for API communication, you'll typically use the DateTimeFormatter.ISO_INSTANT format
println zdt.format(DateTimeFormatter.ISO_INSTANT)
// 2016-04-12T19:20:45.539Z
// some more examples of formatters
println zdt.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME)
// 2016-04-12T14:20:45.539
println zdt.format(DateTimeFormatter.RFC_1123_DATE_TIME)
// Tue, 12 Apr 2016 14:20:45 -0500
println zdt.format(DateTimeFormatter.ISO_ZONED_DATE_TIME)
// 2016-04-12T14:20:45.539-05:00[America/Chicago]
// you can also create a custom formatter
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("MMM dd yyyy GG")
println zdt.format(formatter)
// Apr 12 2016 AD
</code></pre>
<h4 id="date-manipulation-is-fluent-and-intuitive" tabindex="-1">Date manipulation is fluent and intuitive. <a class="header-anchor" href="https://dantanner.com/post/javatime-for-impatient/">#</a></h4>
<pre><code class="language-java">import java.time.LocalDate
import java.time.temporal.TemporalAdjusters
LocalDate localDate = LocalDate.now()
println localDate
// 2016-04-12
println localDate.plusMonths(1).withDayOfMonth(1)
// 2016-05-01
println localDate.minusMonths(1).with(TemporalAdjusters.lastDayOfMonth())
// 2016-03-31
</code></pre>
<h4 id="prefer-to-work-in-utc-if-you-can-logs-database-timestamps-consistency-helps-avoid-mistakes" tabindex="-1">Prefer to work in UTC if you can - logs, database timestamps…consistency helps avoid mistakes. <a class="header-anchor" href="https://dantanner.com/post/javatime-for-impatient/">#</a></h4>
<p>In a startup class: <code>TimeZone.setDefault(TimeZone.getTimeZone("UTC"))</code>
or as a JVM flag: <code>-Duser.timezone=UTC</code></p>
<h4 id="when-working-with-a-point-in-time-always-be-aware-of-timezone" tabindex="-1">When working with a point in time, always be aware of timezone!!!! <a class="header-anchor" href="https://dantanner.com/post/javatime-for-impatient/">#</a></h4>
<p>Even when working with objects like LocalDate, you must be timezone aware if you’re using a point-in-time operation. And
by point-in-time, 99.273% of the time I’m referring to the now() method.
For example, given the following scenario:</p>
<p>The default JVM timezone is <code>UTC</code>.<br>
At <code>2016-04-12 3:10 PM</code> in <code>America/Chicago</code>, println <code>LocalDate.now()</code> will return <code>2016-04-12</code>.<br>
At <code>2016-04-12 6:10 PM</code> in <code>America/Chicago</code>, println <code>LocalDate.now()</code> will return <code>2016-04-13</code>.</p>
<p>If you’re in Chicago, that looks like tomorrow, and might be a bug for what you’re trying to do! The code is doing
precisely what it’s told though; the date in London is April 13th at the time the LocalDate is created for the UTC
timezone.</p>
<p>So to repeat, when working with specific points in time, be aware of the timezone you’re working with. For example, if
you want to set a date 30 days out from now according to your business location, you could
say <code>LocalDate.now(ZoneId.of('America/Chicago')).plusDays(30)</code>.</p>
<h4 id="further-reading" tabindex="-1">Further Reading <a class="header-anchor" href="https://dantanner.com/post/javatime-for-impatient/">#</a></h4>
<p>The <a href="https://docs.oracle.com/javase/8/docs/api/java/time/package-summary.html">java.time javadocs</a> are actually really
good, so read them for more detailed information. Wrapping up the whirlwind tour, here’s a quick table to help get you
started with a few objects and their example uses:</p>
<table>
<thead>
<tr>
<th>Object</th>
<th>Example Usage and Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td>LocalDate</td>
<td>Birthday, Contract date where no time or timezone is needed</td>
</tr>
<tr>
<td>ZonedDateTime</td>
<td>Most points in time, like some startDateTime, endDateTime</td>
</tr>
<tr>
<td>YearMonth</td>
<td>When you only want to work with a year/month combination. Less commonly used, but handy for date comparisons if you don’t want granularity to the day. e.g. Credit Card Expiration Month</td>
</tr>
<tr>
<td>LocalTime</td>
<td>e.g. Chris’s hardware store opens at 8 am. This date is irrespective of timezone (i.e. you wouldn’t change the opening time when daylight savings rolls around.)</td>
</tr>
</tbody>
</table>
Dynamic Grails Tomcat datasource configuration with Etcd2015-04-30T00:00:00Zhttps://dantanner.com/post/dynamic-grails-tomcat-datasource/<p>Ever wonder if you could modify a Grails datasource while the app is running?
Probably not, and that's totally fine...most people don't need to. We had a couple reasons though:</p>
<ol>
<li>During a disaster recovery situation where a non-clustered database goes down, you want to point all the apps at a failover database. By default this means you have to update the config and restart all the apps. On a typical AWS instance, this means at least a minute of downtime for a bigger Grails app. Not the end of the world, but not great.</li>
<li>One of our databases is a catalog of product information that can be drastically changed. We wanted to be able to clone the catalog, apply massive data changes to it (this can take a minute or so), and then point all the apps in the cluster to this new database without downtime. And we also want to be able to revert to the old database if something goes wrong.</li>
</ol>
<h4 id="first-question-how-can-you-change-a-tomcat-datasource-while-the-app-is-running" tabindex="-1">First question - how can you change a Tomcat datasource while the app is running? <a class="header-anchor" href="https://dantanner.com/post/dynamic-grails-tomcat-datasource/">#</a></h4>
<pre><code class="language-groovy">package com.foo.util
import groovy.util.logging.Log4j
import org.apache.tomcat.jdbc.pool.ConnectionPool
import org.codehaus.groovy.grails.commons.GrailsApplication
@Log4j
class TomcatDatasourceUtil {
static void ensureCurrentDatasources(GrailsApplication application, List datasourceNames) {
log.debug "Ensuring datasources are current"
datasourceNames.each { String datasourceName ->
ConnectionPool connectionPool = application.mainContext.getBean(datasourceName).targetDataSource.targetDataSource.pool
def dataSourceFileConfig = application.config."$datasourceName"
// discover the properties we want to potentially change. if changed, update and purge the pool
List propertyNames = ['url', 'username', 'password']
if (propertyNames.any { String propertyName ->
connectionPool.poolProperties."${propertyName}" != dataSourceFileConfig."${propertyName}"
}) {
connectionPool.poolProperties.url = dataSourceFileConfig.url
connectionPool.poolProperties.username = dataSourceFileConfig.username
connectionPool.poolProperties.password = dataSourceFileConfig.password
connectionPool.purge()
log.info("${datasourceName} was modified and refreshed")
} else {
log.info("${datasourceName} didn't change")
}
}
}
}
</code></pre>
<p>Grails version 2.3 and onwards uses the <a href="https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html">Tomcat Connection Pool</a> as its datasource provider by default. If you're not using Grails 2.3+ yet, you're probably using the Apache Commons DBCP, and can switch by using <a href="http://grails.org/plugin/jdbc-pool">this plugin</a>.</p>
<p>Basically, you pass the <code>ensureCurrentDatasources</code> method your grailsApplication and a list of datasource names you want to inspect for changes and potentially refresh. The datasource name(s) are typically defined in your DataSource.groovy. e.g. If you only have one datasource, it'll be named "dataSource". If you're using multiple datasources, they might be named "dataSource_auditing" or whatever you've specified.</p>
<p>The method is implemented to compare the current Tomcat connection pool values for the username, password, and url against the current Grails configuration values. If any settings have changed, it'll update those connection pool settings and call the purge() method in the connection pool. purge() will basically perform a graceful reset of all the connections so that they establish their next connection with the updated configuration. I chose username, password, and url because those are the things that we might change. There are more properties in the pool that you could possibly change, but you probably don't want to change much else, since there is some critical state being managed by some of the properties.</p>
<p>OK, so you know a way to dynamically update a datasource while the app is running.</p>
<h4 id="next-question-how-should-i-wire-in-this-dynamic-update-capability" tabindex="-1">Next question: How should I wire in this dynamic update capability? <a class="header-anchor" href="https://dantanner.com/post/dynamic-grails-tomcat-datasource/">#</a></h4>
<p>The short answer is, whatever works best for you. Here's the path we went down...</p>
<p><strong>The initial approach:</strong>
Our application has the following attributes:</p>
<ul>
<li>It uses an inline plugin where we keep our domain classes and services.</li>
<li>We use <a href="https://saltstack.com/">Salt</a> to manage our external config files</li>
<li>It uses the <a href="https://github.com/bluesliverx/grails-external-config-reload">External Config Reload</a> plugin to allow us to dynamically update the app config when we change the config files.</li>
</ul>
<p>With those attributes, we initially implemented a hook into the TomcatDatasourceUtil by defining the onConfigChange event in our plugin's Config.groovy, like this:</p>
<pre><code class="language-groovy">def onConfigChange = { event ->
TomcatDatasourceUtil.ensureCurrentDatasources(
application,
['dataSource', 'dataSource_auditing'])
}
</code></pre>
<p>This worked fine, but seemed like a clunky solution. For the catalog database update scenario, the application essentially needs to remotely communicate with salt, so that salt could remotely update all of the application's configuration files. We keep all our salt configurations in source control, which didn't really fit the model of what we wanted to do.</p>
<h4 id="the-better-approach-or-at-least-this-has-been-working-well-for-us-so-far" tabindex="-1">The better approach...or at least this has been working well for us so far: <a class="header-anchor" href="https://dantanner.com/post/dynamic-grails-tomcat-datasource/">#</a></h4>
<p>Rather than use a tool to constantly push out config file changes on the fly to our cluster of apps, we thought it would be better if we inverted the technique...i.e. have all the applications get their configuration from a central location. This is where <a href="https://github.com/coreos/etcd">etcd</a> comes in. The summary of etcd is that it's "a distributed, consistent key value store for shared configuration and service discovery with a focus on being simple, secure, fast, and reliable."</p>
<p>You can run just about any groovy code in your Config.groovy and Datasource.groovy. So rather than have the application get its datasource config info from a file, we have it load the datasource URL from etcd. e.g. Here's a snippet from our external config file:</p>
<pre><code class="language-groovy">import groovy.json.JsonSlurper
dataSource {
// most properties are directly set
pooled = true
// ...
// the url is retrieved from etcd...make sure the etcd resource is properly protected
def jsonSlurper = new JsonSlurper()
def catalogUrlConfig = jsonSlurper.parseText(new URL("http://etcdlocation:2379/v2/keys/dataSource/url").text)
url = catalogUrlConfig.node.value
}
</code></pre>
<p>This will take care of your app getting its initial url value from etcd. You can put whatever else in etcd that you want...for our case we only need to dynamically change the url.</p>
<h4 id="so-now-how-do-you-update-the-datasource-for-a-cluster-of-applications" tabindex="-1">So now how do you update the datasource for a cluster of applications? <a class="header-anchor" href="https://dantanner.com/post/dynamic-grails-tomcat-datasource/">#</a></h4>
<p>In your application's Bootstrap.groovy init, make a call to a class like this:</p>
<pre><code class="language-groovy">package com.foo.config
import com.foog.util.TomcatDatasourceUtil
import grails.plugins.rest.client.RestBuilder
import groovy.json.JsonSlurper
/**
* Application service to get and set values from a centralized remote configuration service.
*/
class RemoteConfigService {
def grailsApplication
public static final String CATALOG_DB_URL_KEY = "dataSource/url"
protected static final int INITIAL_RETRY_TIMEOUT_SECS = 5
/**
* Runs a process to watch for configuration changes to the plancatalog datasource URL.
* If the URL value changes, call the datasource utility to update the connections to point at the new database.
*/
void watchForChanges() {
if (!grailsApplication.config.remoteConfig.enabled) {
log.info "remoteConfigBaseUrl not configured - will not watch for config changes"
return
}
log.debug("watching for changes")
Thread.startDaemon {
int secondsToWait = INITIAL_RETRY_TIMEOUT_SECS
while (true) {
try {
String url = get("${CATALOG_DB_URL_KEY}?wait=true")
if (url) {
grailsApplication.config.dataSource_plancatalog.url = url
TomcatDatasourceUtil.ensureCurrentDatasources(grailsApplication, ['dataSource_plancatalog'])
}
secondsToWait = INITIAL_RETRY_TIMEOUT_SECS
} catch (Exception e) {
log.warn("Exception occurred while watching for config changes. Will wait ${secondsToWait} seconds and continue watching", e)
Thread.sleep(1000 * secondsToWait)
// double the length of the time to wait before retrying, up to a maximum of 30 minutes
secondsToWait = [60 * 30, 2 * secondsToWait].min()
}
}
}
}
/**
* Get the configured value for the given key
* @param key
* @return the current value
*/
String get(String key) {
def jsonSlurper = new JsonSlurper()
def json = jsonSlurper.parseText(new URL("${grailsApplication.config.remoteConfig.baseUrl}/${key}").text)
return json?.node?.value
}
/**
* Set the given key to the given value in the centralized configuration service
* @param key
* @param valueArg
*/
void set(String key, String valueArg) {
RestBuilder rest = new RestBuilder()
def resp = rest.put("${grailsApplication.config.remoteConfig.baseUrl}/${key}") {
value = valueArg
}
if (resp.status != 200) {
throw new Exception("Error setting configuration value for key=${key}. Response=${resp.body}")
}
}
}
</code></pre>
<p>This is a very basic implementation of an etcd client that can watch for changes, update the grails configuration upon change, and also allow the app to update an etcd value. There are more robust etcd clients available, but we didn't need (at least not yet) the added dependencies and complexity.</p>
<p>It's pretty fun to watch once you get it all working. Essentially this is the flow:</p>
<ol>
<li>A cluster of grails applications start up, configure their datasource URL using the etcd config, and watch for changes.</li>
<li>Some time later, one of the applications clones the database, makes changes to it, and then sets the new URL value in etcd.</li>
<li>All the applications are then notified of the updated etcd value and dynamically update their datasource to point at the new URL.</li>
<li>"dataSource was modified and refreshed"!</li>
</ol>
<p>Does this actually work?
Yup - extremely well. I actually expected the initial implementation to be a little brittle (e.g. maybe needing a more robust etcd client), but it didn't need any changes in the 6 months I watched it in production.</p>
Groovy collect vs spread-dot operator2014-11-15T00:00:00Zhttps://dantanner.com/post/groovy-collect-vs-spread-dot-operator/<p>Yesterday I was doing some Groovy code cleanup with the wonderful <a href="http://codenarc.sourceforge.net/">CodeNarc</a> static
analysis tool. One of the violations it found
was <a href="http://codenarc.sourceforge.net/codenarc-rules-unnecessary.html#UnnecessaryCollectCall">UnnecessaryCollectCall</a>.
The summary of the rule is "<em>Some method calls to Object.collect(Closure) can be replaced with the spread operator.</em>"
e.g. Replace <code>things.collect { it.name }</code> with <code>things*.name</code>, or even <code>things.name</code> if what you're after is a property.</p>
<p>But when I performed that refactoring and ran all the tests, some failed! Here's why:</p>
<p>I made the mistake of assuming that the spread operator behavior is always identical to the collect method. For a *
<em>non-null</em>* collection, it is. e.g. The following code will produce the same result regardless of the technique you use:</p>
<pre><code class="language-groovy">def things = [ [a: 1], [a: 2] ]
things.collect { it.a } // returns [1, 2]
things*.a // returns [1, 2]
things.a // returns [1, 2]
</code></pre>
<p>But if the collection you're operating on is <strong>null</strong>, the three techniques will result in different outcomes:</p>
<pre><code class="language-groovy">def things = null
things.collect { it.a } // returns []
things*.a // returns null
things.a // throws a NullPointerException
</code></pre>
<p>What this means is that if you're working with Collections that can potentially be null, you need to think about the
consequences of the dot operations before using them. i.e. Don't ever use the implicit spread operator (<code>things.a</code>) if the
collection can be null. And only use <code>*.</code> if it's ok for the result to be null.</p>
<p><em>tl;dr;</em> Explicit and implicit spread operations are great, but be aware that they are less forgiving than the collect
method.</p>
Playhouse2010-10-16T00:00:00Zhttps://dantanner.com/post/playhouse/<p>A few months ago, I promised Maria that I would get her and Daniela a playhouse for their birthdays. I was originally going to buy them a little Step2 playhouse, which would've been totally fine. But my good friend Rob, a skilled woodworker, convinced me that building a playhouse was the way to go. I was a little tentative about the idea, since I hadn't built a real structure before on my own, but he very generously offered to help (and with two kids of his own, meant that his wife Karen also very generously helped...we still can't thank them enough!)</p>
<p>So I bought this <a href="https://dantanner.com/playhouse/playhouse-small.pdf">plan</a> (which I can't find any more, so am including it here), made a few modifications, and off we went:</p>
<p>Starting with a pile of boards...</p>
<p><img src="https://dantanner.com/playhouse/100828_IMG_3919.JPG" alt=""></p>
<p>and a friend who know's what he's doing...</p>
<p><img src="https://dantanner.com/playhouse/100829_IMG_3929.JPG" alt=""></p>
<p>things started to take shape, and a lot of space!</p>
<p><img src="https://dantanner.com/playhouse/100829_IMG_3931.JPG" alt=""></p>
<p>We passed the initial kid inspection...</p>
<p><img src="https://dantanner.com/playhouse/100829_IMG_3933.JPG" alt=""></p>
<p>and had lots of "help".</p>
<p><img src="https://dantanner.com/playhouse/100829_IMG_3937.JPG" alt=""></p>
<p>The first wall goes up...</p>
<p><img src="https://dantanner.com/playhouse/100904_IMG_1129.JPG" alt=""></p>
<p>And the playhouse is now fully operational.</p>
<p><img src="https://dantanner.com/playhouse/100918_IMG_1257.JPG" alt=""></p>
<p>But we realized that it absolutely has to be red.</p>
<p><img src="https://dantanner.com/playhouse/100918_IMG_3944.JPG" alt=""></p>
<p>Much better color!</p>
<p><img src="https://dantanner.com/playhouse/100919_IMG_1305.JPG" alt=""></p>
<p>I finished up the railing in the dark with a million mosquito friends.</p>
<p><img src="https://dantanner.com/playhouse/100920_IMG_1307.JPG" alt=""></p>
<p>One of the owner-operators.</p>
<p><img src="https://dantanner.com/playhouse/100929_DSC_4771.jpg" alt=""></p>
<p>Of course it needs a doorbell.</p>
<p><img src="https://dantanner.com/playhouse/101003_IMG_1369.jpg" alt=""></p>
<p>and a solar-powered light for late night parties.</p>
<p><img src="https://dantanner.com/playhouse/101016_DSC_4783.jpg" alt=""></p>
<p>With the trim, it's pretty much done!</p>
<p><img src="https://dantanner.com/playhouse/101016_DSC_4780.jpg" alt="">
<img src="https://dantanner.com/playhouse/101016_DSC_4782.jpg" alt=""></p>
Domain names for daughters2009-06-29T00:00:00Zhttps://dantanner.com/post/domain-names-for-daughters/<p>Though the biggest domain name purchases are probably long-gone since the initial dot-com boom, they still remain a valued property. They'll never be as valuable as real land of course, since we can always create more of them, as opposed to things like lake-front property.</p>
<p>The recent mad rush for facebook profile names reminded me of this new global scale of gold rush activity. And yes, I'm a little annoyed that someone that has been farming this interweb as long as I have was offered a name like dan.tanner2 (in my day, we used Mosiac. And we liked it!). It also reminded me of something I'm hoping will come in handy for my two daughters some day - their very own short and memorable domain names.</p>
<p>Finding a decent available domain name is a tenaciously difficult PITA. It's an exercise in creativity and luck, and girls names are especially tough. Not only do you have to wrestle with the general lack of availability, but also the likelihood that they'll change their name when they get married. In my case, that's sometime after age 28, when they've agreed to start dating.</p>
<p>In my case, my girls are named Maria Mae and Daniela Teresa. All good variants of maria and daniela were taken, which would've been ideal. And no last names were considered because of the potential marriage name change. After tinkering around for a while, here's what I ended up saving for them:
<code>mariamae.com</code> and <code>danielat.com</code>
They're both fairly mnemonic, and they're both flexible enough to last. And they're both shorter than my domain name!</p>
<p>It'll be a few years, but hopefully they'll be useful. Not to mention give me a little boost in the cool dad category. Not that I'll need any help in that category.</p>
<p>Now...where did my favorite plaid pants go?</p>