Wednesday, July 29, 2015

Fighting with Java SSL and Confluence

The Error:

Connection test failed. Response from the server:; nested exception is javax.naming.CommunicationException: [Root exception is No subject alternative names matching IP address found]

Some notes:
* Does not happen when using java 1.8.0_45 (Java 8 u45)
* I ran into problem when using java 1.8.0_51 (Java 8 u51)
* Running Atlassian Confluence 5.8.4 on EL6

Updates To Follow:

Monday, January 5, 2015

What do you mean "it's in production"?

What do you mean "it's in production"?

Short Story: 

To many groups use the word "production" and that word changes meaning and risk depending on the group in question.

Long Story:

Our usage of the term "production" leads to some issues as it changes context based on audience. Operations can look at "production" as a matter of state where as Development may see it as a function or environment. To confuse matters worse Ops may also refer to it as an environment given it's history of working with Dev.

An example of the issue is demonstrated by a common statement.
Jane: I show server prodX is down, whats going on?
John: It's ok server prodX is not in production.
Jane may be reasonably confused by Johns statement. What does John mean by the server is not in production?

  1. "prodX" is not in the production environment. (Maybe the node name is mislabeled or misunderstood.)
  2. "prodX" is in the production environment but is not in a production state.
  3. "prodX" is is not in a production state and is not in a production environment. 

This also applies to the simple statement.
The code has been deployed to production.
This could mean:

  1. The code is servicing customer requests.
  2. The code is located in the production environment.
    1. It is servicing customer requests.
    2. It is not servicing customer requests.
  3. The code is not in a production environment but it is taking requests.

From an Ops perspective there are three options for any given service outage:

  • SEV1/2/3: Drop everything (Severity determines response time)
  • SEV4: Don't wake me I will get it when get in.
  • REQ#: Nothing is broke you should send in a request.

Operations service response for State \ Environment
Not Active \ Non-Prod => REQ#
Not Active \ Prod => SEV4
Active \ Non-Prod => SEV4
Active \ Prod => SEV1/2/3

Developers on the other hand have a near reverse perspective.

  • P1: Project is in active development.
  • P2: Project is waiting on resources.
  • SEV#: Help to make sure the application keeps working. There are constraints on what we can do.

Developers response for State \ Environment
Not Active \ Non-Prod => P2
Not Active \ Prod => P1
Active \ Non-Prod => P1
Active \ Prod => SEV#

When you merge the views you will see that there is a conflict for Not Active \ Prod, Active Non-Prod, and Active \ Prod.
In the case of "Not Active \ Prod" and "Active \ Non-Prod" the Ops teams will give low priority for supporting resources to the Development teams. This can impact speed of delivery of fixes and features to production but it conflicts with Ops immediate role of keeping things working. Likewise because the hands of the Dev teams are usually tied in "Active \ Prod" environments the Dev teams are slow to help seeing that it is Ops job to control those environments, even though it is the previous chain of work that feeds production.

How does DevOps resolve this issue?

There are two issues with understanding "what is production".

  • How do you deal with scope and work priority?
  • How do you deal with semantics?

How do you deal with scope and work priority?

In some ways DevOps flips the priory of both Dev and Ops. The problem area is what gets DevOps focus and it is where Developers and Operations must meet. The mission for each group stays the same, however structure needs to be added to have the groups work together in those contentious areas.
The Ops team needs to understand the work of the Dev's. They need to see the features and be active in understanding why a function is monitored or not monitored, what is the impact of a missing function, and what are the business drivers for a service. All of those things help in determining risk which Ops deals with regularly. The Ops team doesn't change how they respond to SLA's for Prod and Non-Prod but they should work with the Developers on seeing what is happening in those space.
Devs need visibility into what Ops is doing and dealing with. From a service perspective they need access to logs, monitors, and trends which should all be jointly reviewed by Ops and Dev as it may directly impact Dev's mission. Both groups need to create a constant feed back loop that helps push each team to better work quality and ultimately better service for the business.

How do you deal with semantics?

The issue of semantics is difficult. That "Sami Language of Norway, Sweden, and Finland have a 180+ snow and ice related words. This is needed because the distinctions are important. The more I see of companies dealing with this issue the more I think the same of ITSM and DevOps. However I do not now what that word should be or how it should be structured as both state and environment are important to IT, but both had different context for different groups.

Thursday, September 4, 2014

Learned something new on my way to testing https posts

This is a quick post for testing https posts. If you were doing something simple and just sending http posts then you could use netcat (nc) with something like
$ sudo nc -l 80 < resp_200.txt
The file "resp_200.txt" simply has a a line "HTTP 200 OK". Netcat will open port 80 and respond to what ever connects to it with the text from that resp_200.txt file. It will dump output to the screen with the http post it received. Nice way to test your post. Ah but what do you do when you are sending a post to HTTPS?

OpenSSL can be used to provide some netcat type functionality. You can see a detailed view of this from Wsec "USING OPENSSL AS A NETCAT REPLACEMENT".

Quick how to is:

Create self signed cert
$  sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem

Now use openssl to make a listener on port 443

$ sudo openssl s_server -accept 443 -cert mycert.pem

In my case I'm using Ruby to post to https similar to this example on Stackoverflow but with HTTPS instead of HTTP.

After you post you will see a bunch of text showing you the HTTP post information you sent.

This is a nice way to test your post code before you start hitting your production site.

Friday, August 8, 2014

Time to look at Ansible

I find it time I take a look at Ansible. From the ad-hoc perspective it seems to fit in nicely and probably works a little better than my "batchcmd" bash script I use to run commands across a list of hosts.

I am not sure I agree with Ansible's design concept. The marriage of configuration management and ad-hoc execution is prone to problems. Puppet Labs and R.I.Pienaar, creator of Mcollective, take a pretty strong stance of trying to avoid execution of scripts or code because some crazy things can happen. It's not so bad when it's one or two systems but when you do something on hundreds and they run into the problem that could be an issue.

The nice thing about automation is it enables you to do good things faster across many more systems.
The problem with automation is it enables you to do bad things faster across many more systems.

Both Puppet and Ansible are declarative in nature, so they do not require the item to change only that the item "becomes" a finished state. However given Ansible's "push" philosophy it is also looking for "immediate consistency". This may work for small deployments but in larger systems this becomes problematic as I can break everything from the start. Puppet follows the "eventual consistency" model which when properly accounted for leads to large scale services that deploy as opposed to small scale ones that for whatever reason will not get to the state you want when you think you want it. It also gives me an idio-second to change something back because I broke the first 5 nodes that checked in and not all 100.

Ansible does have a "pull" option which does allow for "eventual consistency" but this begs the question of "why have another configuration management system?". Which then just brings us back to what does Ansible give me that Puppet does not? At this point it gives me the ability to run ad-hoc command execution across multiple systems. Puppet already gives me configuration management and Mcollective gives me safer orchestration.

After trying Ansible out I may change my mind and there is nothing that fundamentally says you cannot use both tools. Ansible being agent-less has it's advantages. However puppet agent has saved me from my stupidity when I broke SSH and OpenSSL. It was nice to have Puppet correct my screw up after locking everyone out of SSH. I could see where it would be nice to have Ansible save me when I inevitably break Puppet doing something silly.

UPDATE: 2014-08-11T19:33-50
Ansible allows you to --ask-sudo-pass to prompt for your sudo password on the systems. This means that as long as your user has sudo rights to run the command in question you can do what you like. Not sure how --ask-sudo-pass stores your password though?

Thursday, July 24, 2014

Too many or too few STEM degrees?

Are there too many or too few STEM degrees and are they being utilized correctly? Infoworld's Patrick Thibodeau and Sharon Machlis note that "For 74 percent, STEM degrees lead to non-STEM jobs"

In my opinion geography plays a huge part in many people's reason to stay. Some of this depends on specialty but for the most part the coasts are looking to decrease their cost and trying to find degrees in other locations. However there are many cases where people simple do not want to move to the coasts. People with STEM degrees are not stupid generally weigh the benefits of moving to California, Texas, or New York / DC.

I have had several colleagues choose to move to the east or west coast. Some who have even moved back after a stent because they simile didn't like something about those places. There is also the cost of living that can change drastically. Trying finding 10 acres or 5 acres of land near San Francisco, Seattle, or New York. You won't not with out having a sizable commute. You can do that in the Midwest (well not Chicago) and to some degree in Texas, but even Texas is starting to show strain due to traffic. As much as I complain about traffic in Kansas City it is nothing compared to Texas, San Francisco, New York, or Seattle.

I have seen a number of tech workers wishing to be able to work remotely. This has strong pull to many but it is a new paradigm that many companies are not equipped culturally or technically to handle. However that is changing. I have seen startups and some large name places coming to understand that they can do work remotely and this will remove the geographic issue to some extent. In the past I have told several companies that I would love to work with them but I am in no position or have no desire to move to their city / state.

This of course has it's own trade offs. Companies in the Midwest are just now realizing the career options that companies like Google, Linkedin, Facebook, or Amazon offer to STEM employees. Otherwise they have been traditional limited in advancement options and frankly most people I know that are successful in STEM are generally drive to want to advance.

In short STEM will see better utilization when geography is removed from the equation as geographic markets play a huge role in the decision to move to the various STEM "meccas" in the US.

Monday, June 23, 2014

RedHat Subscription model and why OEL is easier.

DISCLAIMER: Some colleagues got me started on this rant, and make no mistake this is a "rant", so I fully intend to blame them for this lengthy deluge of vehemently worded information.

I often work on keeping our repository of various Linux distributions available for my job. Tool's like cobblerd are a great help in this effort. By having our repositories readily available and being able to automate much of our deployment makes deploying Ubuntu, CentOS, and OEL systems a snap. I can have a new system built from scratch in less than 30 minutes and I can do many of them in parallel. (NOTE to self, really really need to start looking at RAZOR)

Now this is all fine and dandy but there is something missing from this setup, Redhat or (RHEL). Why you ask? RHEL repositories require you to be subscribed to them in order to get the updates. You cannot mirror them like you can mirror Ubuntu, CentOS, or OEL. If you want to have a local mirror you are ardently encouraged to use Satellite server. Which is great until you start figuring out the cost of having the privilege of mirroring Redhat's package services. There are other problems with this process, mainly that the mechanics of registering a license on a server with Redhat is an automation headache. It is very possible that I am missing something in this process but frankly I do not see anything that allows this to be automated because I have about 6 billion licensing combinations to go with. (Yes, that is an exaggeration because this is a rant). The astute among you will think well, Redhat has come up with using the answers file to make this process work automatically... Please note this requires Satellite.
Satellite sucks.

"But why do you think Satellite is so bad?". First the cost of Satellite is something like $10K. This isn't much for an enterprise and you would be right, it is not. I have no problem with the cost of the server itself, but wait their is more. $10K gets you the server but you need another subscription for each server you have managed by it. It's not a little bit either. Last quote I had was close to $200 to $250. I hope price has changed and I know the advent of the virtual machine and having RHEL be your Hypbervisor has mucked with how things are but really? $200 per server + $10K for Satellite + Hardware needed to run it locally + Subscription = I can pay for a good Sysadmin and do it all with CentOS or Ubuntu or Debian. This has not even touched on it's usability or lack their of. Redhat would do well to hire a UI designer and process engineer to stream line the workflow for managing it's systems.

I want to like Redhat. I really really do but OEL hosts the current repo for you to mirror. It's licenses are cheaper and it's basically the same as RHEL. Yes, Redhat made it first and Oracle just added their secret sauce for Oracle stuff but OEL is like CentOS but with support. Oracle's Linux engineers are some decent guys and I find them fairly easy to work with. (Oracle, your application support sucks. Sucks so bad I would probably only call support if I was drunk. Regretfully I don't drink so it makes your support process very hard.)

Redhat has great people but they really need to figure out this pricing and licensing/subscription game. As it stands now it is more pain than it is worth using.

Thursday, November 29, 2012

Dell XPS 15 with Ubuntu

There are some exciting things going on at Dell and Canonical. Project Sputnik is going well and one of my comrades at work is thoroughly enjoying his Dell XPS 13 with Ubuntu 12.04 on it. As he says "it just works". He's not a big fan of Unity but he's getting old and questions change (^_^ love you man!). I don't mind Unity. It's getting better but that is a discussion for another time.

So Sputnik has done something that I have been dying for, they have looked at getting the Dell touch pads working sanely. They have pretty much accomplished this with the XPS13. However the XPS13 is a small laptop. Great portability and power for it's size but frankly not something I would consider a work horse like my Dell E6420. I like the larger screen and tend to have a lot going on in various windows and work spaces. For single minded tasks the XPS13 is great and works great for running two terminals side by side but doesn't fit my needs. So I am tackling the XPS15 (L521x).

Meet the Dell XPS 15:
The XPS15 is a rather powerful slim laptop. It has various configurations for purchase of which I have the high end Enterprise version but I am ordering a high end small business lappy as well due to my needs with VMs. In short more cores better option than faster cores.
NOTE: If you look at the Dell website you have to be careful as the XPS 15 is listed under large business and small business both of which have different processor options. The large enterprise version offers the i5-3320M and i7-3520M where as the small business version has the i5-3120M and i7-3612QM. (See comparison at There is also the consumer/home version is the XPS 15Z so make sure you look at all locations when you are looking for the one you want.
Overall Anandtech has done a nice write up on the small business version and I would suggest giving it a read.

Ubuntu and XPS 15.
Ubuntu 12.04 "Precise Pangolin" installed with out a hitch. Recognized wireless right away and had no issues with the Intel 4000 gpu. Working with the Nvidia GeForce GT 640M gpu was a littl more involved but I will get to that in a moment. Overall the Ubuntu install was flawless, fast, and free from major issues like I have experienced before. That said there were some small issues that were quickly remedied by having the right drivers.

One big issue that you will probably notice right away is that the track pad is on ludicras speed. It took some steady slow fingers to go through some menu's. The track pad issue was resolved thanks to Project Sputnik mentioned earlier. The XPS 13 and 15 essentially use the same mouse drivers so I added the ppa for Sputnik and this made the track pad much more resonable. The pad does take some getting used to as it is a full click pad and does away with the middle button. Depending on you mouse settings you may need to use one finger clicks to do left click and two finger click to do right click. Oh and almost forgot, two finger scrolling works both vertically and horizontally  This was a feature lacking on my E6420 that could only do vertical scrolling with two fingers.

Your next major hurdle, and it is time consuming, is dealing with Nvidia's Optimus technology. I was a little ticked at first about having to work with it and that the bios doesn't let you disable it like previous version did. However the technology has progressed and so has Linux thus making Optimus viable and helpful. What is Optimus you ask? "Optimus" allows non-3d intense apps to run on the local gpu while programs needing more power can use the Nvidia card. Thus power demand decreases based on usage. It also helps for Linux because most the Intel graphic drivers just work, especially when adding external monitors.

To get Optimus to run in Linux you currently need an application called "Bumblebee" which adds Optimus support to Linux. Ultimately Bumblebee will be included in the kernel but until then it is a separate project. Once installed there are some configuration things that need to be done all of which are covered in "Bumblebee's" documentation/FAQ for how to install it. Once in place you should be able to run "$ optirun {command} " and that application will render using the Nvidia card instead of the Intel 4000.

One gotcha I ran into is that you essentially have to remove "bumblebee" and install it again should the kernel change. Hopefully that will all be taken care of in future versions.

Pain Point and Deal Breakers
There is some setup that you need to do to get Ubuntu to run correctly on the XPS15. It is a little more than what you would do with E6420, however it is doable and once done the system works well and I had few issues. However, the two issues I did run into are show stoppers.

While typing I would often send the mouse flying or inadvertently click on something. After using the XPS15 for some time I switched back to my E6420 only to suddenly realize why I had such issue. The XPS15 essentially centers the larger trackpad on the "H" key thus shifting the normal location over and with the added size means it is constantly near my palm. If you use proper typing techniques with the home keys then hitting the 6,7,y,h has the strong potential of hitting the trackpad and causing an inadvertent click. The "disable as you type" feature helps some but not enough.

The second issue is with the heat generation. This comes in two forms; cpu and gpu, both caused by pushing the graphic's capabilities of the system even slightly. Minecraft and League of Legends work well on my E6420 but slaughtered the XPS15 due to heat. The only fixes I found for this was a firmware updated that essentially clocks the GPU down which invalidates any reason to purchase such hardware.

On paper this system has great CPU, GPU, and memory capabilities. Regretfully it doesn't survive the real world which is to bad because I really wanted to like this laptop, but given the track pad placement and heat issue I will have to wait till the next model from Dell to see if the correct the issues.