My evening in telecom – BSA Post-Mortem

The term Post-Mortem is typically used to describe the process of evaluating a body, after it has been disconnected from whatever it is that causes life to function, or a project once it has come to its end. Today is no different, and it feels like a bit of both.

Over the last decade I have been involved in a shit ton of projects, most of which are no longer even able to be pulled up on the way back machine, or were kept under the radar and never released. I tend to land a lead type position, not because I am the most knowledgeable or skilled, but because I am a no nonsense guy who wants little more than to solve problems. Don’t let me sugar coat it, I am a muckraker; when I see process or flow problems I attack them, and I have learned to do that ruthlessly. This energy tends to be accepted by people I’m fighting for and for those I’m fighting with, it is… well.. a mess.

One project that ended up being a thorn in my side was World at War, primarily because of personality conflicts and an all around unwillingness to solve problems with process and flow, but also because of a lack of planning. In the end I offended a huge swath of our “community” because I suggested that developers should be held to a higher standard, which led to a ton of infighting and further personality conflicts.

The reason I am telling you this is that I was just released from my duties as a Business Systems Analyst in the telecom industry, where I have just worked for 12 days. The following are the cleaned up and truncated notes for my experiences. Enjoy your look behind the curtain.

The First Day

I walked into telecom bright eyed and bushy tailed at 8am. I spent the first 2 hours of the day filling out the typical HR quizzes that affirm that I wont be a hazard for employment (Harassment, Company Policy and agreeing not to do anything stupid). After I was finished with the paperwork Michael, a colleague, walked over with a project.

Telecom, as an industry, is very complicated. There is a lot of information flying around, and that information is being manipulated by tons of people and results in a lot of fur flying. Most of the work is handled via one off Excel documents, and most of their data are forecasted and accepted dates for project management. There are a number of phases, and I was asked to focus on one of the major bottle necks – order generation.

Within this company orders are created by construction staff, management and otherwise, who are technically supposed to be out in the field. These orders are tremendous, typically dealing with six figures worth of materials and services, which have to be submitted to and reviewed by the client before being purchased. A single order may take a number of hours to complete. Estimates range from 2-3 of these being completed in a day by any single person. A certain level of experience, education and visualization are needed to be able to fill one of these out, so they cannot be offloaded to a vendor or admin, mind you. These orders are created from an excel document. You may not catch the emphasis on that, so I’ll reiterate – These orders are created from a SINGLE excel document, which is duplicated, saved with a given title for lookup after the fact, and then their quantities are filled out. This document is a real monster, thousands of rows in length, with vendor or client part numbers intermingled. I was witness to a training for some of the other new employees and it was a real test of will to even get through it. Pages of cabling, antennae and other materials go on and on.

After spending about 2 hours with the guys creating these orders and looking over a pretty reasonable 5% solution provided by Michael I was already thinking about how to solve the situation:

  • Of course this sounds like a web based application,
  • maybe some jQuery.Suggest type interface to quickly populate the parts you actually need,
  • some form of historical tracking so similar sites can just be duplicated and then customized to fit requirements or modifications to the site,
  • some form of audit tracking to allow not only the ability to review what orders have been submitted but also to allow for SOX compliance as the application grows to serve its eventual needs,
  • reporting can be hooked in to allow for managers and staff to see the actual improvement in time by using this method of generation, common components and services or things that are just never used

The list of possible things that can be done, short order, to improve the pipeline, even in a small way, is lengthy. I was eager to get this off to someone for guidance within this new ecosystem and so I reached out to the head of the development team, Evan:

Good Day Sir,

All is well on my end and I have my first project in the mix, and a few questions for you. I don’t have access to my email or laptop yet, but I just completed a quick overview of the ordering process and I think it can both be looped into a single web (intranet) based application. First and foremost I’ll need to know how your configuration management system works, what database engines are available for me to use for development and what type of server environments are available for me.

What kind of formal documentation are you looking for and are you going to be managing my day to day workflow or do we have weekly meetings to catch-up/plan next steps.

I will probably have more questions in the next day or so.

I am making this team automation crazy.

Let me know if you have any questions for me. I’ll get back to you as soon as possible.

Hope all is well.

.:Bob

About an hour later I received the following:

It is way more complicated than you may see at first. I think we need to chat before you automate anything. Often if deep enough though is not put into it – automation can extrapolate problems down the road.

– Evan

Hmm… well, instead of giving you all of the email thread to devour over the next week I’ll sum up the rest of the day with the following:

  • I am asked to fly out to meet the development team in person, they don’t live anywhere near local.
  • I have a class to take so I cannot go out for any contiguous block of time.

You may have noted a couple of oddities off the bat. I don’t have a laptop or access to my company email yet. Typically these things are ordered well in advance and setup for the person to begin using as soon as they walk in the door – a worker without his tools is not a worker at all. I mean hell, if I don’t bring pencil and paper to class I am considered not ready for class and can be kicked out.  Also, you will note that the questions I asked were pretty straight forward – what have we got, how do I get you information about projects coming from my end and who am I reporting to? The responses were not nearly as interesting, kind or informative as they should have been. Luckily I brought my bad the fuck ass Alienware laptop and have a platform through which to be able to actually keep in contact with them, using my personal email.

It is now noon and the group takes me, and a few other new hires, out to lunch to celebrate. I get back to the office and am quickly off to pick up my phone. Evan calls to fill me in on how things work. Here are a select few paraphrased quotes:

  • If it weren’t for me telling the VP that he needed you, you wouldn’t have this job.
  • You are a smart guy and I love that, but do you know what the problem is with smart people? It is hard as hell to subordinate them.
  • My market does not represent the rest of the nation.
  • These people will never leave excel, no matter what I try to suggest.

Odd. The lead developer gets to take credit for my having a job in a company that needs developers like a fish needs water? Who the fuck tells a guy who is smart that he is supposed to be subordinate to anyone in a knowledge based industry? Why does it matter what my market is in relation to the whole nation if I was hired on to serve this market?

I actually stood up in the room at this point and asked the construction staff if they were tied to Excel in this fashion. Not a single person replied with anything other than [fuck excel].

I don’t like where this is going, but I’ve made a commitment.

I go home, happy as a jay bird. I have a job, it isn’t really meeting the goals or impression I was given but besides Evan, the people I work with are pretty damned smart and know how to get their shit done.

Day #2

Today is going to be better.

The day starts off similar to the previous one. I am on my personal hardware using my person al email to follow alerts and details. I get pulled into a few not quite so interesting, but informative, meetings and piece together some more of the picture for solving the problem. A phone meeting is set for 2pm with Evan to piece together the situation for this ordering system, discuss ideas and get some direction. Evan has added Dave to the invite list. Dave is a developer. We should be able to talk shop.

The meeting is on GoToMeeting, and includes visuals from them and confusion from me. These guys have been building a tool into excel to help automate the process. It consists of a new company branded ribbon bar with two buttons, one to pull in an item master listing, currently residing in a google spreadsheet, and one to submit that system into the ordering system, built on top of OpenERP.

Sure, this is solving a part of the problem, but guess how many rows there are in this imported spreadsheet? And better yet, why are you submitting it directly into the ordering system? I share my thoughts, after watching them work with their current system for a couple of days, but my thoughts fall on deaf ears. I am quickly aware that I am fighting a losing battle. These two guys point me at the item master listing as the primary issue, suggest that careful solutions that comply with SOX are more likely to get traction and then go on to let me know that I don’t know what I am talking about, that my market is not representative and that I need to make sure to run any ideas past Evan and Dave.

Okay, well I thought I was doing that. I walk over to my managers and share my experiences. Their grimaces and frowns tell me that this is not going well. Clearly my idea is not being accepted because they aren’t letting me get it across to them. It is time for a formal proposal. I sit down with my manager and he walks me through the whole process from end to end that he wants this application to do and so … I’m off to my desk to piece together a formal proposal of how I would like things to be pieced together to get off to the developers for input, hopefully this time before they pass judgment upon it.

On my way to my desk I pass up one of the guys who handles ordering, Bill, and so I ask him to give me a quick overview of the items master situation. Clearly if he is ordering things they have to exist somewhere. He raises his eye brows and assures me that

  1. the items don’t change frequently, and when they do we are able to find out about them pretty quickly
  2. All of our vendors part numbers are able to be converted to a client part number because our clients need to have that information anyways
  3. Any discrepancies are able to be checked against their online databases

Day 3

I worked with helpdesk to get access to my email, so I’m now using web mail and have a shit ton of mail to go through. Mind you, I sent out the first request, as did 3 other people, on day 1 within the first 10 minutes of walking into the office. But, what ever, I’ve got access to my fucking email biznatch!

Cue the scary and ominous music. Today starts off with being handed 30 pages of HR shit that I need to sign, fill out and submit by 3pm to be able to stay on and work any more days per federal law. Well, I guess this is going to happen.

I start blazing through paperwork and note that my I9 needs to be filled out. The form requires that you submit two forms of identification, or your passport, to be able to show employment eligibility. Well, sucks to be me, but now I get to inconvenience my wife and ask her to take two hours out of her busy work day to bring me my documentation and drive home. Meanwhile I am doing what I can to keep my arm from going numb. In between meetings and orientations and getting pulled into a meeting with my managers to get further notes together for this application idea I am filling out paper at a feverish pace. In the middle of this tremendous stack of paper is an authorization to be a broker at a pawn shop called Jewelry-Plus… No Shit.

Didn’t sign that.

In my down time I am in contact with Dave through chat. Here are some of the details:

  • This is a new shop, only a year old
  • “we have no time for documentation. Everything was due yesterday”
  • Another guy is responsible for documentation and intranet development
  • Dave is an open source guy, python + java.
  • Dave thinks excel is a lowered standard
  • Quickbase is the platform that all of the data is housed in, and was a “major win”
  • OpenERP also being used
  • These guys hate leaving excel
  • They are looking to eventually move to a custom platform
  • There are only 6 IT personnel
  • He thinks that the items master is the largest problem, ordering second

This is when things get fun. Apparently the web application idea wont work, because in a perfect world the construction staff are on site filling them out. They need an offline solution. I stand up and ask the guys building this about where they are when they are working on an order and if they typically have access to the web when doing so – every single person has a smart phone and they all freely admit that they are all going to be tied to their chairs through the end of march just to get through the orders for the sites we are already responsible for, let alone the ones that are added as the projects scale and our client gives us more business. I forward this along and again I’m stepping on toes.

Dave: How can you say that? I know for a fact that the managers and officers don’t have access to the web for days a time.
Me: That may be the case, but at least in my market, these guys cannot even make their services listing until they get into quickbase. You can always allow them to work in excel locally, but ultimately they need to get that information into quickbase.
Dave: Why not have excel do that for you?
Me: The solution I’m piecing together is to allow them to do their own thing, work however they want and ultimately submit through our auditing and validation system. If you stick it in a web application they are no longer going to have to submit multiple orders that get kicked back, the whole validation process is able to be handled internally. As things evolve, such as government regulation and required components or other business logic driven elements, you can add them with a data driven promotion or an application update and let it filter through. It provides a single point of failure and control
Dave: That is a cultural change. Those don’t go over well. People just wont use it. Your approach to show them something that will take a 6 hour job and turn it into a 10 minute job just doesn’t work. We have been trying to push solutions for tracking and ordering for the last year and they just wont leave excel. People can continue to do their jobs and not get fired for not using the new system we are pushing. Why would they want to learn something new?
Me: Fair enough, those are valid points, but the sales point for automation and web solutions is always going to be the return on time. I am looking at 10 guys who are going to be data entry clerks for the next two months.
Dave: They don’t care about that. I think you thinking we don’t realize that.
Me: These guys do.
Dave: That is one market. What you are saying wont fly with Evan.
Me: Yes, and I have a green light to build this. I have been brought on to solve problems and this is just the first one. Maybe we can talk it through and make it more palatable?  I am not looking to make huge waves.
Dave: But you are. You have one idea, and that’s it. Web Apps.  You aren’t open to anything in this particular case other than a web app.
Me: I have many ideas, I just want an appropriate solution to a problem domain, which I have been asked to solve. If building this into quickbase is the route that is where it will be. It doesn’t have to be an enterprise level custom application.

It is about noon when I run through the checklist of things I need to send out and note that one of the things that has to be included is a mandatory drug screening so I had to jet. I get out to the screening lab and sit down when Evan calls, check the bullet points here:

  • If it weren’t for me telling the VP that he needed you, you wouldn’t have this job.
  • You are a smart guy and I love that, but do you know what the problem is with smart people? It is hard as hell to subordinate them.
  • My market does not represent the rest of the nation.
  • You don’t seem to be understanding things.

Fun fun! All around the mulberry bush… now I’m not understanding things! Yay!

When I get back over to the office I go into the office with the guys that are my managers and Peter, a national level project manager, and I voice my frustration. Not since WAW have I ever been told I don’t know what I’m doing, don’t know how to solve it and that I am making waves. I was under the impression that we were a single company, but this is borderline childish behavior and I need some assistance straightening things out.

Peter: Bob, you are a regional level developer. You work with this market and are only going to be building a set of solutions for this market. You report to this guy right here and nothing else is going to go on.

Michael: Let’s jump on a call to see if we can get this sorted out.

Unfortunately everyone that has been difficult to understand so far are in meetings. I get back to my desk and a fourth guy reaches out, John. John calls me and after a couple of volleys of my history and his we get down to brass tacks:

John: So, when can you come out to meet up in person?
Me: I can jump on a plane any time. but I do have a class that meets on Tuesdays and Thursdays, so I can’t do contiguous days until school is out.
John: Well, I was hoping to get you out here for three or four straight days to get your oriented and what not. Ill have to talk this over with Evan. I’ll be in touch.

One of my managers, Alan, forwarded me an email from a higher up:

Alan,

Please prioritize Bob’s initiatives for the local market but, he must get approval and by off on the technical solution from our systems team (Evan) so, we make sure we are adhering to our corporate standards and the solution would integrate with the companies goals. We don’t want him to build solutions which go against what we have already initiated or work that is in progress.

Eric

I thought things through, and it was clear that I had stepped on toes and was pissing a lot of people off – Although I couldn’t, and still cannot, explain why. In order to straighten things out I sent out the following email to Evan:

Hello Evan,

I want to apologize, I think I may be stepping on a few toes and would like to curb that before things get too saucy. At this point, I am going to defer to your guidance and would like very much if we can sync up and I can take on a couple of tasks to start off with. You’ve been in the industry for a while now and have the best understanding of what the market requires.

I spoke to John earlier and he said he would reach out to let me know what day is good for me to come out to meet you in person.

Enjoy your weekend!

.:Bob Chatman

Day 4

I walk into the office and bright and early again with a shiny new email in my personal mail inbox from Evan:

Software, Scrapper and Company Data

Bob,

 

In interest of company policy and practice, I would like for you to meet with John today to hand over any Software, Scrapper and or Company Data that is not housed or facilitated on company servers or hardware.  I need to ensure that any Web Apps in place have User Authentication and an Audit Trail for all field and record creation or modification so we are SOX compliant, within company policy and legal as our parent company is publicly traded.  Documentation would be ideal.  Lastly if you are not comfortable with us having full control and ownership of these services, software and or scrappers then I ask that they be stopped so there is not misrepresentation as to their potential misuse.

He is referring to a web form I created to help one of their employees. It is a questionnaire that sends an email off to the compliance department. No databases are involved, no records are created and no data is being scraped. I have written other utilities in the past that would qualify as crawlers and scrapers but this is not one of them.

I send John a zip of the form, cc a few interested parties and get back to work. in the background a firestorm has ignited and things are not getting better. Evan calls me mid day to let me know the following:

  • If it weren’t for me telling the VP that he needed you, you wouldn’t have this job.
  • You are a smart guy and I love that, but do you know what the problem is with smart people? It is hard as hell to subordinate them.
  • It is my job to tell the guys in this market that not only are they not going to get this app to solve their problem, they are not going to get an application to solve their problem.

I asked for an audience with the guy who gave me the project initially, Michael, trying to explain to him the situation. I told him about the shit that had been going down and he was outraged. he called Peter and Eric, the VP of Program Management, looking for assistance and so I forwarded my experiences along to Peter who was quite concerned. I was asked to forward along my initial email with some details:

Bob,

Please put together:

1. a detailed description of the ordering tool that I am trying to build (including the excel samples that initiated the project);

2. what the interface will be;

3. the function the ordering tool will serve (i.e. push to quickbase, generate orders, generate pick tickets); and

4. the data/system/backend access you need.

Once the scope is clearly defined I’d like to set-up a call with you and the systems dev group to make sure we are all on the same page.

Thank you.

Okay, well I had already had some input regarding the ordering tool proposal I’d drafted and while it was approaching 18 pages now (with user stories and database table ideas) it was no where near what I would consider complete. I discussed it with Michael and decided it would be more than enough to get us at least to have a proof of concept type system to be okayed.

In the mean time I reached out to Evan for clarification on the compliance form. I was assured that there was not a problem with the form staying up until whenever they found time to activate it within the intranet. Of course I got echoes of the above bullet points reiterated, with an odd sounding addition.

  • If it weren’t for me telling the VP that he needed you, you wouldn’t have this job.
  • My market does not represent the rest of the nation.
  • You don’t seem to be understanding things.
  • As of Friday we had decided we weren’t going to work with you. I am rethinking that for now though.

I suck it up and send off a response to Peter’s email, including Alan, Eric, Michael, and Evan in the response:

Hello Peter,

1. a detailed description of the ordering tool that I am trying to build (including the excel samples that initiated the project);

The current state of the proposal is attached. The UI Mockups have been left off because I have not had much time to invest in this so far. I can probably have them to you by the end of the evening if they are needed. The zip contains the excel files that initiated this process.

2. what the interface will be;

The system can be built a number of ways, but would likely be the easiest to maintain as a web application or desktop/mobile application that checks for updates at startup/each day. The attached specifications assume a web application but are far from set in stone.

3. the function the ordering tool will serve (i.e. push to quickbase, generate orders, generate pick tickets); and

This is primarily a tool to generate the initial order to be reviewed. Once the orders are generated and reviewed by one of the managers it is possible to speed up the process of uploading that information into quickbase or OpenERP, assuming such facilities are in place or can be developed.

This tool is also a single point of applying business logic or rules to the process. Applying certain requirements or double checks to the process, such as ensuring that an outdoor site only has appropriately categorized parts applied to it, and enabling us to apply site specific rules such as a first carrier project needing specific components. Currently the staff have to continue to hand this information down through word of mouth or trial and error.

As far as I know, Michael may be able to clarify, there is no plan for this tool to submit any actual orders or to generate any tickets. There is an allowance for uploading to a system in the proposal but that portion has not been well sourced or defined.

4. the data/system/backend access you need.

My application requirements in the document are admittedly vague and depend on what infrastructure we have in place already. Everything in this document is up for discussion, debate and correction to fit our needs.

If there are any further questions or comments, I have a clear schedule for the rest of the day and have my phone with me.

.:Bob Chatman

All Items were attached and a meeting was scheduled for 9am the following Thursday (Day 6).

Day 5

Thankfully it is a quieter day. I don’t hear from the development team, but I am taking a more conservative approach to things. Reporting and tracking requests are being generated and handed to me, but instead of responding I simply pass them off to Evan for guidance and input.

Day 6

6am is early, but it is even earlier when meetings are being shifted around. Evan adds Dave to the invite list and pushes it out to 10am, which conflicts with Michael being able to attend. I shift the meeting and accept that things are going to get messy off the bat. I am on my laptop, notes in hand for what needs to be covered and it falls apart almost immediately.

A flowchart was created with scrappers and data flowing all over the place. Items we can order, items we want to order, items that exist and items that we can bill for each with the same three bullet points and a scrapper connecting it to the next set all the way along. My heart sank more and more. I was cut off more times than I could count, had my name repeated back to me in a fashion that only children have done previously, with the term mom, and my proposal which is clearly written and succinct is being misquoted as generating picks, orders and needing to be a web application.

The four times I calmly asserted that it was not the case, and that I don’t think they have read the same proposal that I have written, I was strongly told – That is not at all what you have written here.

My proposal was compared to an employee requesting a sundial app on a mobile phone… surely it is possible to scope out all of the work needed to pull it together, but they do already have a digital clock in their hand. My proposal was compared to an abacus, because surely they can use the application that I’m suggesting or they can just use the tool that is already in place.

Four separate times Evan was asked if he had a problem with me working on [some straw man simplified version of the proposal] and four times he came back with no, except on principle.

I got off of the phone and had to head out to class. The compliance manager reached out to ask me to make sure I got in writing that the form didn’t have to move. and so I sent out the email:

Evan,
Just to tie this up,
per our conversation last week this can stay up until further notice,
correct?

only to receive the following:

I understand that as a company we have many needs.  My position within the company has been to allow people to be resourceful in ways to do their job in better ways for the benefit of the company as it relates to data; however, when it comes to software and databases things get a bit more complicated.  Meaning, I encourage being resourceful but if these solutions relate to these two components then I am obligated under company policy and data laws as it relates to publicity traded companies to ensure we comply with the following:

Is the solution managed under our hardware – if not is it authorized ?
Who owns the software?
Have we taken legal action with the owner or creator to secure contracts such as NDA, Professional Services Agreement, W9.

Has compensation been discussed or officially denied as to prevent legal issues.
Do we have user authentication – SOX compliant and Bill 198 compliant?
Do we have field definition information and or documentation to the software or data system?
Who are the administrator of that system?
Are they employees and authorized to be admin?
Is our data encrypted?
Do we have an audit trail for all fields and all users?
How often is our data backed up?
Has the solution been architected in such it can be absorbed into other systems for role-up?

Again,  I like the fact that you and Bob put the tool together.  If Bob is interested in meeting compliance with the tool and delivering a timeline, I am fine with it in place.  If not we cannot address the matter until 2nd quarter as we are very short staffed and the solution will have to be handled outside of software or database management.

Evan

And this

Bob,

Please do not initiate/continue the use of any scappers without clearance from legal and systems development approval.

Thank you

Peter

I think you can see where this is going.

I was asked to step back and just put the POC together at this point, so I started trying to figure that part out. My laptop arrived on Day 7, so I went to town. In my naiveté, not having further correspondence meant things could get done. I was working with groups on specifications for further reports to hand off to Evan, answering technical problems before they were even asked for and working with the team around me pretty god damned well.

Day 8

Today started out pretty swell!

No crazy angry emails to have to sort through. No arguments or pride issues to climb out of. It was just an all around reasonable day to be a developer. Eric had come in for a few meetings and asked me to get let him know before I took off so we could sync up. I worked, got lunch, worked some more. He came over at about 4:20pm and we went over to an empty office to sync up. He had his pad of paper, and so did I.

As we walked he said he wanted to discuss my job description and how things were going so far. Then I closed the door, we sat down, he explained – They really stretched the definition of Systems Analyst for my role. They had originally wanted someone to be a project manager, and my skillset is more technical. “And so, we are going to terminate you as of today. here are some things i need you to sign for acknowledgement. i have your final paycheck and … did you have any keys?”

He handed me an acknowledgement, handed me a notice that health care would be cancelled as of this day, and my final pay check, escorted me over to my desk where he watched me pack up my personal laptop and escorted me over to the common area where he smiled, shook my hand and said “thank you for your time, do you carpool? oh, okay. it is okay if they leave now as well.” and walked off.

Aftermath

Now, of course… I recognize that some of this is company policy and regulation. But as I look back on this and look back at my job description, I was hired to be a bulldog and the quotes above are nothing like the level of ridiculous I could have been. Most everything had a star next to it saying I was looking for further input and evolution of the ideas. Most everything had a nice little cross over it requesting assistance and information about the internal systems.

As of right now, I’m looking back on that post I made a week ago and I’m disappointed in the outcome. There is clearly a problem here, considering the amount of energy invested on just proving me wrong,

This is one crazy one off job where I am clearly not being used to the best of my abilities. Everyone who I was fighting for thinks I am a simplified martyr, the targets of my frustration are the key master and door keepers who, on numerous occasions, fanned the fires of a hostile workplace and left me to smolder.

Of course, I am not going to lose sight of reality.

I still have my 4chan, Minecraft, Node.js and UnCodeX2 to cheer me up!