“Value ++”
One of our engineers on the TechOps team coined that term. It references the shorthand operator for “increment” in various coding languages. It also is a motto for what we should be doing as a team… always adding value.
Here are a few things in our day to day operations that have been a serious “value –”
- answering JIRA tickets that have no description
- “customer has issue B” .. but both customer name and issue details are omitted from that sentence
- doing things manually, over and over again
At LogicMonitor, most of the tasks requested of the TechOps team come in the form of JIRA tickets. A new application may be ready for deployment. A customer account requires a rename. We also have to deal with operational tasks like moving new customer accounts from demo to production environments.
Because LogicMonitor is rapidly growing, we are always trying to be more efficient by automating ourselves out of work. We decided to automate parts of our DevOps tasks thorough AWS Lambda functions, API calls, and JIRA tickets. This allows the team to not only keep track of existing tasks that show up in our queue but also spend their time doing more important things.
“Value ++”
Projects and issue types
In order to differentiate tasks from other items, we first had to lock down specific JIRA Projects and Issue Types, creating a separate issue type for every task we wanted to automate. This not only makes things easy to organize, but it allows us to lock down who can or cannot make specific tickets.
In this particular blog, we’ll go over one of our simpler use cases: automatically performing an account rename.
The simple stupid
This crude Lucidchart (below) shows the basics of what we did. A Lambda function is triggered by a CloudWatch Event rule set to run every 5 minutes. The function will make a JIRA API call to retrieve a list of tickets. Using those tickets, we will grab the necessary information and make subsequent API calls to backend services within LogicMonitor to perform specific actions… such as renames. Lambda will also actively update and close the tickets upon completion of the task. The first thing we need to do is know what tickets to look for.
Query JQL from Lambda
JIRA Query Language (JQL) is one of the most flexible ways to search for issues in JIRA. We use a JQL query with the JIRA REST API in order to find specific open tickets with issue types of “account rename”. This should return a list of associated tickets.
endpoint = "https://jira_url/rest/api"
jql_issuetype = "issuetype='Account Rename'"
jql_project = "project='TechOps Request'"
status = "status=Open"
jql = ("jql=" + jql_project +
"+AND+" + jql_issuetype +
"+AND+" + status
)
r = session.get(endpoint + "/2/search?" + jql % locals(), headers=headers_jira)
response = json.loads(r.text)
for issues in response["issues"]:
customer = issues["fields"]["customfield_10001"]
target_name = issues["fields"]["customfield_14673"]
Taking the list of open tickets, we need to be able to glean important information out of them, some of them in the form of custom fields.
Custom fields
Custom fields are created by users and not found in JIRA by default. For our specific use case, we created a few fields such as customer name, target name and rename date. From the code example above, you can see that within the JIRA API, you can not specify just the name of the field, you’ll need to add a customfield_id.
Pro tip…if you don’t want to look at a page of ugly json, you can also go into the advanced JIRA search bar and type in the name of the field.
Event driven computing… most of the time
Usually, when we build apps on Lambda, we have components like Lambda functions and event sources. An event source is a service in AWS that publishes events that would be processed by code from within a Lambda function. In this case, performing a rename upon JIRA ticket creation could have been handled with a post function and an API Gateway. However, customers have their own maintenance windows and preferred times for an account rename to happen. Sometimes a customer may want their account rename to be done Saturday at 4am… during my personal maintenance (sleep) window. We decided, as a workaround, to use a cloudwatch event as a lambda scheduler.
today = datetime.datetime.today() - datetime.timedelta(hours=7)
desired_date = datetime.datetime.strptime(issues["fields"]["customfield_16105"].replace("-0700",""), "%Y-%m-%dT%H:%M:%S.%f")
if today > desired_date:
create_rename(customer, target_name)
Our Cloudwatch event would run every 5 minutes, triggering our lambda function. The function will first check if the current time has exceeded the value we parsed from custom field rename date (see code above), and only then will we allow the function to continue.
Putting it all together
At this point, we have collected the information we need. We are able to make API calls to backend LogicMonitor services to perform the rename (that code won’t be shown in this blog). However, we also want to treat the JIRA ticket as a state file. We don’t want to keep grabbing the same open tickets over and over again. This is where we want to use another JIRA API call to move the ticket to a different workflow step (eg from “Open” to “In Progress”). However, just like custom fields, we need a specific transition id, which you can find by editing your existing project workflow. We can now update the status of our JIRA ticket programmatically:
def changeStatus(key, id):
jira_request = {"transition":{"id": id }, "fields": {"resolution": {"name": "Done"}}}
endpoint = "https://jira_url.com/rest/api"
r = session.post(endpoint + "/2/issue/%(key)s/transitions?expand=transitions.fields" % locals(), data=json.dumps(jira_request), headers=headers_jira)
return r.text
Saving people from people
Customer renames for the team used to be an extremely arduous task. Looking back at the Confluence revision history for our account rename runbook is akin to cleaning out your basement after 20 years. In addition to being very time consuming, there was a mish-mash of processes that involved stopping puppet and running, for some reason…a ruby AND a bash script. There was sometimes an application restart required, but not always. As we grow in size, the only scalable solution is to automate repetitive, manual, and often mind-boggling tasks. It not only allows us to provide better service for customers but also gives us opportunities to bypass the mundane to embrace the innovative.
One last tip – and this is the most important part – when we want to automate anything that requires manual input from other people, we have to take human stupidi… uh… error into consideration. Make sure to create validators and conditionals to combat this.
Plus, witty warning messages are a “Value ++”
