What I Learned On The AxIon 2 DSA Project, A Cautionary Tale

This post isn’t going to be easy to write.  While this project was a success for the company, the ramifications for me did not end well.  I made mistakes, more social than technical.  The technical mistakes were quickly fixed.  The social mistakes cost me my job and basically ripped my life in pieces.  This hasn’t been the first time that this has happened for me, but each time it gets worse. When things are worse than having a Schizophrenic for a boss, something has gone really wrong. I’m writing this because I feel that I need to.  I feel that I need to have my say about this.

The Axion2 DSA was my major project at PerkinElmer, my last employer.  While I did not have the title, I was the principle mechanical engineer for both the Axion2 and DSA.  The projects were done under very tight schedules with small room for mistakes and complete on short timelines.  Since I was “restructured apparently both the Axion2 and the DSA have done well, though without access to the sales numbers I can’t tell how well.  This report is hard for me to write because of what happened during and after the events. The  fact that I’m still highly stressed out about what happened is telling, considering that I have been in a series of employment periods where things went belly up.  This one went very bad even by my standards.

I do think that engineers should get a look at how bad things can go into the toilet even when the project succeeds.  It used  to be that engineers were respected and even had jobs in positions of authority.  Those days are long over.  These days engineers are more than likely to be kicked around and kept doing regulatory compliance paperwork than being allowed to design  great products or invent new processes. If this is what your job is like, you are not alone. It’s become the norm in our “fundamentally transformed” society.

http://www.americanlaboratory.com/914-Application-Notes/154453-Direct-Sample-Analysis-Time-of-Flight-Mass-Spectrometry/

http://www.perkinelmer.com/catalog/product/id/mz320025

I’m going to start with a little history and background,  Names will be replaced with initials to protect the guilty.  I started work in July 2008 at what was then a small company that was owned by CW.  The company made mass spectrometer components, built some prototypes and essentially lived on the patents from electrospray liquid chromatography mass spectrometry(LC/MS) and the fact the company made all the glass capillaries for LC/MS instruments made by the industry.  The one thing that CW and the little company had never done was put an actual mass spectrometer on the market.  There had been a constant stream of attempts, but they had, for many reasons, failed to come to fruition.  This will become important later.

Just before I hired on, three important events started.  1. CW signed a long term contract for the sale of a TOF mass spectrometer as part of a more complicated instrument.  2. CW settled some disputes relating to IP issues with the large university down the road.  These IP issues had apparently been around since before CW took over the company.  3. PerkinElmer was brought in to actually manufacture the TOFs for said contract since  the contractee was not confident that CW’s company could manufacture the instruments and became interested in purchasing CW’s little company.

Which is where I entered the picture.  Most of my career had been doing various contract jobs for a variety of companies for the past 15 years or so interspersed with working at my local hardware store between contracts. I had worked in a laporascope manufacturer, a large government physics lab, an aircraft sensor company and an interferometer company. I interviewed and took the job no knowing anything about mass spectrometry and essentially expecting to just do engineering changes and part design, which is typical contract work.  As the work for the TOF and other projects grew I was offered a full time position which I accepted even though the pay was less than I wanted.  Remember that this was late 2008 and I logically didn’t want to be out in the cold.

In 2009 I did a number of various projects mostly revolving around the Flexar 300, where I essentially repackaged the electronics and redesigned the ion guide plugin among other things. I also designed prototypes for other projects that CW and other scientists wanted. In all of this I was essentially a minor engineer with no real responsibility for the overall fate of a project. I just took the assignments from our group manager, MD and completed them.  And I didn’t have a lot of contact with CW, who was tied up with the legal issues, the purchase of his company and fooling around with a project that he had had going with a friend who would come up with money from time to time. Here’s a video on the Flexar.

Perhaps the most important thing about 2009 is that the purchase meant that for the first time in his life,  CW had a boss and deliverables he HAD to meet.  That meant changes that I’m not sure that CW really understood.  I don’t think that he realized that he couldn’t do things the way he had always done them.  He now had things like approval processes and budgets to deal with. He’d never had to deal with the restrictions placed on projects by costs and deliveries because the historical revenue sources of his little company hadn’t relied on having to deliver mass spectrometers to market.

Another significant event at the end of 2009 was  the bringing in of US as a leader in the quadripole MS. This gave CW another  heavyweight to compete with. Which had it’s advantages and disadvantages.

In 2010 the emphasis switched to the TOF for the most part. This was driven by the customer requirements for a lower cost instrument, performance issues,  and the fact that the instrument was not even close to being production ready.  I worked on a variety issues involving various issues in the vacuum system.  At this time my largest frustration was that many of the projects I was handed didn’t seem to have a goal or requirement that they be finished. I would be handed something, get to the point where I was ready to finish and then I would be doing something else. One problem was that MD had more or less given up the manager’s  role to do more project work and we lost the discipline to get things done.  Also many of these projects were fairly speculative in nature and sometimes things don’t work out.   In research, this isn’t all that unusual and I didn’t really think that that was a serious issue. And there were the fires that kept bursting up.  I should have paid more attention to those issues because the clues were all around me were not really working as well as they could. The fact that some of the work I was doing was resolving issues that needed to be resolved  and got done may have obfuscated the real issues.  And anyway a job is a job, was my thinking.

Wiki page on TOF mass spectrometers.

http://en.wikipedia.org/wiki/Time-of-flight_mass_spectrometry

Toward the end of 2010, CW dumped a new TOF on us.  This was the result of after hours designing on his part and seeing what others had done at the shows, ASMS and Pittcon.  To be honest I thought  that the new TOF was going to be a great machine and a good project to work on.  What I didn’t understand then was that CW hadn’t gotten a budget and approval for this new instrument.  So my office mate, AL and I did a  whole bunch of work that was not approved.  To be honest, I don’t think that CW really understood at that point that he needed to get project approval.  And frankly upper management didn’t impose the kind of discipline on operations that was really needed. I think that the largest problem was the manager, SI, placed in charge of our little circus, was a sales guy and we didn’t have anything ready to sell. Something that CW didn’t seem to know how to change.

That brings us to 2011.   AL and I, spent the first couple of months working on the new TOF2 and MD worked on the DSA, with PSC working from a prototype that had been brought over from our office in Shelton.  Now a major issue had become evident with the TOF that was used in the customer’s instrument.  The internal rack computer was being obsoleted and this was important because the TOF required the bus in that computer because a custom control/ data transfer board had been created to work in that type of bus. Our large customer also wanted significant price cuts and cost down.

Now I wasn’t in the meeting, but I heard what happened.  CW tried to sell his new TOF2, a machine that had not even been prototyped and was essentially a completely new design as the solution to the cost down.  As might be imagined it did not go over well.  I expect that CW didn’t really understand that he now had to get budgets and  timelines approved for something like that and that using customers as development resources was not going to work once an instrument was  in production.

SI put his foot down finally. The DSA would get first priority and the TOF would be upgraded.  The TOF2 would be put into hiatus until we actually had a working TOF or forever.  The problem was I think that SI meant the DSA That MD and PSC were working on and not other projects.  But CW took it upon himself to start development of a DSA with fancy robotics and complicated motion of the sort that might be used in a machine for integrated circuits that was not something in any way needed for the DSA.

And I got the TOF dumped in my lap.  I’m not sure how that ended up.  MD was involved in the DSA and US’s stuff, AL was doing source work and sheet metal for the TOF,  VG was working on Flexar with US and well I guess I was the only one left.  In any case we had a brainstorming session, came up with some ideas and away I went. I knew That I had about six or seven months to get the new machine ready, or at least on the test stand before the computers ran out, depending on shipments and the need to revamp large parts of the vacuum system.  But everything was doable if there were no major issues and I didn’t see any. There was very little room for error and none for messing around.  I thought I was in good shape.  I didn’t factor in CW.

So I started in.  I pushed for a task list from CW to have guideline, wrote it up on the whiteboard and prioritized based on the task that was likely to involve the most work or time.  In a project I try to choose the largest task first so that the body of the work will be done first with adjustments for lead time.  In the case of the TOF it was a new one piece flight tube, the reflectron, the plug in and the feedthroughs to the pulse and detector areas.  Actually the feedthroughs were an ongoing project that I had already been working on for several months off and on since early 2010.

I got the new one piece flight tube done and quoted out, the changes in the reflectron, started when in charges CW, yelling and screaming at me about the top of the reflectron where a shipping clamp block was installed to ensure that the reflectron didn’t  bang around during shipping.  The verbal assault went on for a few minutes until I firmly got CW calmed down and went back and forth about this very minor issue.  This would become a repeated pattern.  I think it happened about six times altogether. CW would come in and verbally assault me over some trivial issue or another until I got him calmed down and resolved the issue I called it being his “whipping boy.” I later realized that that was exactly the case.  CW was using me as a stress reliever for the presentation he would be making later in the day.  The fact that stressing me out was costing him my time and health never seemed to occur to him.  Or the fact that only the idea that I could work past him was keeping me in my seat.

Anyway I moved on to the plug in, designing changes to the ion guides, the sweeper, the capillary exit lens and adding the trap door so that the capillary could be pulled without venting the instrument. I got all the parts designed and sent to the shop.  And waited.  I moved on to other parts of the machine and designed still more things.  And waited.  Now our shop crew was very good, so I wondered what was happening.

What was happening was CW.  He and an intern that was the son of a friend of his were designing up parts for CW’s version of the DSA and because anything  with DSA on it got priority, essentially monopolizing the shop time with iterations of designs, none of which were ever assembled, at least in the time I was there.  So, for at least five months the new plug in drawings sat waiting for spindle time. The one crucial assembly that needed testing and verification.

That brings us to the feedthrough plate. Now of all the issue that I had to deal with this was the most important to this instrument and future instruments.  The fact was that for whatever reason, and nobody ever explained it to me, the connector choices made in the TOF prototype made no sense.  The vendor was not a presence in the vacuum market and as nearly as I can tell the connectors were never really rated for vacuum.  So they leaked.  I got involved in early 2010 when CW wanted to go to higher voltages to increase detector sensitivity and that required a connector change.  I went to the vendor’s website and found one with the rating I needed and put out an RFQ.  They noquoted.  Even though the connector was in the catalog nobody had ever bought one of the high voltage rated connectors in that shell.  Then they downrated the connectors we were using.  Fortunately I had had quite a bit of previous experience with vacuum and hermetic connectors and I went to work.  The problem is that vacuum connectors are, for the most part, welded to their bulkheads to ensure a good seal.  I found some very high voltage connectors for the high voltage connectors and started to look for replacements for the rest.  Now there were weld on 5KV connectors available on a bulkhead, but I could not fine a screw in BNC connector for the detector line that the previous solution, which was more or less a hack was causing issues with feedback.  With a referral from  my high voltage company vendor, I found a weld on connector that would resolve all the issues.  Not only that but the vendor could weld on all the low voltage connectors and give us the plate as one unit.  Now when I worked at the Nuclear physics lab we did this as a matter of course.  It seems expensive when you see the whole part, but you are replacing a bunch of expensive connectors and you don’t have leaks through your feedthroughs.  And the feedback and ringing issues through the detector line go away.  I did a quick cost/benefit calculation and the numbers looked good, so I had a PO put together and had CW sign it, which he did.

The plates come in and they worked.  But I’m walking through the break room in the afternoon and CW is there eating lunch.  He starts tearing into me about how I had failed him.  This was a disaster. The feedthrough plates were too expensive.  We needed to find a vendor to weld them.  All in the most foul language possible. With his mouth full of hoagie. All completely uncalled for.    Remember that I had been looking for solutions to the feedthrough issues for over a year at that point.  I had already looked at all sorts of solutions, none of which worked. Where was CW then? In vacuum cheap is not an option all too frequently.  The markets are too small and specialized.  Frankly by the time you added up the costs of all the old connectors, the machined plate and the assembly time, to say nothing of having to check all those connectors for leaks, I was coming out ahead on cost.  And considering the tight spaces and high voltages involved I didn’t really have a huge plate of options.  The verbal assault was completely uncalled for. I walked  out and essentially started hitting the wall.

It’s at that point that I had MD bring HR in.  Now I don’t know what SI and HR said to CW.  They didn’t talk to me.  They didn’t even bother to get the whole story.  Maybe they didn’t want the whole story. There didn’t seem to me to be any suspension or sanction for an action, that had almost anybody else had done it would have been grounds for immediate termination.  But for CW different rules seemed to apply apparently.  I’m going to guess that they told CW to back off me.  They should have told him to back off on everybody, because I was not the first or the last victim of CW’s abuse and bullying.

So shortly thereafter the company brings in BC to manage our little circus directly so that CW would not be burdened with management duties.  He was supposed to be a heavyweight that would whip us up into shape.  Presumably his job was going to be to keep CW in check.  It didn’t quite work out that way.  I don’t know how it happened, but when CW wants to be ingratiating he can point his powerful personality at you and get you to bend.  It didn’t help that BC’s main goal was to get out of CT and back to HQ as fast as possible.  As he told me, his technique of managing people was not to manage them.  He also took advantage of the fact that our marketing guy quit to take over his role as well.  In any case in the year and a half or so that he was in the office while I was there I think we had exactly one staff meeting and I didn’t get to attend that one because I had just been Restructured.

That was the cap to 2012, a year which started out  me being assigned to be principal engineer on the DSA by BC and them being undermined almost immediately, which created a whole bunch issues in the development process.  To be followed shortly thereafter by a nasty review by BC where according to CW, my output was not sufficient and essentially told I was on probation. My output in what?  I produced 250 odd drawings or revisions and 100 odd part models in 2011 according to a folder I kept for files I entered into the PDM system. The Axion2 was engineered and  would have ready to go by November if I had been able to get the spindle time I needed.  Maybe it was my abuse tolerance that was the issue?

Meanwhile I had to sit in that office and watch CW dither about trivial stuff while the TOF time bomb kept ticking.  As was typical for CW, he didn’t seem to understand that when the clock is ticking, perfection is the enemy.  It was heartbreaking to me to sit there and watch something I had worked incredibly hard to accomplish get tossed away because CW was dithering around the syringe pump door.

So what did I learn from this experience?

One, if you are given the responsibility, make sure that you have the authority to carry out those responsibilities. I should have asked for an official promotion when I was handed the Axion2.  Having “Principal Mechanical Engineer” as the title would have made it clear to everybody that that was the job I was doing.  I also should have asked for a raise to go with it.  That way it would have been clear to everybody that I was the responsible person for the project. I also should not have let CW just throw me off the project like he did no matter how glad I was that it was over. The problem was that once “whipping boy” got into peoples’ heads I couldn’t get the kind of respect that I needed to get the job done properly.  Pushing me around had become a habit and I didn’t have any support to push back.

2. Make sure that resources are available to get the job done.  I should have insisted on spindle time to get the plug in done and get the test stand set up.  Those were big issues and they needed to be resolved sooner so that if there were any large issues they could be resolved quickly. Expecting CW to know to manage his own project was a large mistake. At least at the  “now  we have to make  the instrument produceable” stage. I knew that CW hadn’t been through the process and I knew how he was about long term planning.  It was vital that there was somebody with authority to expedite with issues that came up if need be and I should have at least asked SI about that.  The fact that I only learned that product managers were available later isn’t really an excuse.

3. Make sure that everybody knows what the project schedule is.  I had one, after I demanded and got a task list from CW, with deadlines.  But I never let anybody else know what it was.  Five minutes with an email saying “here’s the timeline, here’s what needs to get done, here’s what I’m going to need”  sent around to the interested parties would have gone a long way.

4. Don’t assume others are on the same page even when they are acting against their own best interests.  Both the Axion2 and the DSA had to be finished simultaneously.  You can’t run the DSA without an Axion2 mass spec to run it on. And our little COE needed to have a product that demonstrated our abilities in the one way it really mattered, sales and revenues.  And the TOF that started the whole thing, the one used in the bigger instrument was a ticking time bomb.  This should have been clear to everybody, yet somehow the urgency didn’t seem to resonate.

5. Keep the job professional.  There was no reason for CW to get so emotionally involved.  His verbal assaults didn’t do anything to change or help the work move forward.  Indeed each of the incidents cost me a day of emotional stress and lost productivity.  And my fear of what may happen if I brought something to him cost opportunities to contribute and make the machine better. The end result was a breakdown of communication in both directions.  The fact that he couldn’t keep it professional hurt both of us.

6.  Make sure that you communicate to the interested parties.  A weekly email update sent around would have gone a long way for everybody to have confidence that the project was moving forward.  Having a log of what I was doing would have given me a hard copy record of what I had done when review time came around.   And the emails would have given me a paper trail as to what I had been doing.

7. When things go toxic, bail.  I should have quit or said I was quitting after the second one of CW’s little tirades.  My role was to get the Axion2 in shape for production.  It wasn’t to be CW’s whipping boy as I called it.  I should have spoken up and if no action was taken, quit and let CW be somebody else’s problem.  The fact that I didn’t want to let my friends, myself and the company down was not enough reason to allow CW to continue to subject me to that kind of abuse. Why should I go through hell so that somebody else gets the credit?

8. Always strive to be excellent.  Sometimes the only thing you can take away from a bad situation is the knowledge that regardless of what happened, you held up your end.  Take pride in that and let the clowns sink in their own swamp.

https://www.linkedin.com/pulse/what-toxic-leadership-andrew-schmidt-phd?trk=prof-post

https://www.linkedin.com/pulse/how-do-toxic-leaders-become-supervisors-first-place-schmidt-phd

While my story is just that, my story, I hope that people can take it as a cautionary tale.  I imagine that stories like this go on throughout corporate America.  The companies seem to have lost focus on what’s important.  You can have stock buybacks all want, but eventually the debt you pile up will have to be paid with real stuff that you can sell.  That’s when people like me should be part of your “core competencies.”  Unfortunately, your BC’s and CW’s will have made sure that the people like me no longer work for you.  They will, between them, poisoned the well for getting work done. Which is bad all around.

This inevitable for companies that are in the grip of Pournelle’s iron law, which so many are.  So they try to buy innovation.  But that’s paying a premium for other’s work.  The value of which is promptly destroyed by the same bureaucratic forces that caused the problem in the first place. Rinse and repeat. Meanwhile nothing is learned or changed because the people who learned what needed to be learned have either left or been restructured.

http://www.jerrypournelle.com/reports/jerryp/iron.html

Do these companies not realize that they are shooting themselves in the butt?  Consider my case.  I was working in the midst of the cutting edge of mass spec technology for almost five years.  I’m a very smart and creative fellow.  Did nobody at my  former employer consider that I might walk into the competition with stuff in my head?  I wasn’t even asked to sign a  nocompete clause in my severance, probably because none of the people doing the layoff had a  clue that a nocompete clause was something that would have been a very good idea in my case.  Considering that I was interviewing with the competition for job two months later a nocompete clause would been something to have.  Of course, had the severance had a clause like that I would have wanted to be paid at least two years salary for the signature.  Considering how everything else in my severance was screwed up who knows what would have happened.  To say nothing of the fact that the company may have had my input on the machines that I was working but it didn’t have the contents of my head.

Which leads to 9.  Never give up all your ideas.  Since the company can dump you at any time keep your best work and creative stuff to yourself until you have resources to develop your ideas,  a budget to finish and your credit is assured.  It also goes without saying that you never sign a contract that says that the company has any rights to anything outside the office.  If they have that clause in your contract ask that it be modified or removed.  If it isn’t, turn down the offer.  The company is quite bluntly making no commitment to you and  seizing every advantage over you that they can get, you don’t owe them anything more than what they pay you for.

Was my experience totally negative?  No, I worked with some good people.  I learned a lot about project management by doing and working with US on his instruments. I moved up the engineering scales at least temporarily.  I have stuff I can show prospective employers that is more than just parts and drawings. But I couldn’t quite close the deal to success.  My greatest frustration is that I felt that finally I was on the  edge of doing something really great, that I had finally left all the skut  work behind me.  It was not  be though.

Even CW probably wouldn’t have turned as toxic as he did if the bureaucracy hadn’t laid off our office mom as an HR person who was redundant.   But CW is probably going to raise the cost of a bad hire because he cost 24 million, the cost to purchase his little company, to hire and it’s going to cost a lot more in probable lawsuits, lost revenue, and poor morale before it’s over.   That’s the company’s problem, not  mine.

In the corporation, bureaucracy determined what was to happen.  It saw things in it’s terms, not mine and certainly not the future of the company’s.  In the bureaucrat think that prevailed I was a liability who had to be removed. A layoff was required.  Why?  I don’t know. To meet some number? And why me? Too creative, too pushy. rocked the boat too much, didn’t know my place, who knows. Was I the nail that stood up in line?  I imagine that in early 2012, the fix, as they say was in and from that point it didn’t matter what I did, the projects I pushed forward or the potential revenues that I would have generated.

That’s the reality in American engineering today.  And just about any field that requires creative thought.  Corporate America has chosen the bureaucracy over every other consideration.  That’s bad. The fact that we creatives don’t even  appear on the org charts as somebody significant is worse. How much revenue did my layoff alone cost the company?  How much would another new instrument be worth?  I had some ideas, but nobody will ever know, now.  I do know that if I, and others like me are not allowed to create, restrained by bureaucratic nonsense, wrapped up in red tape, our time wasted chasing down regulatory requirements stifled by inertia or caught up in malaise, not able  to do our job there’s a good chance that somebody else who wasn’t stuck in the morass will be doing theirs, in someplace far away.  And that there’s always a person like this looking to do his job.

http://www.americanrhetoric.com/MovieSpeeches/moviespeechotherpeoplesmoneydevito.html

And that’s where, if nothing changes in the way we have learned to do business in the last few decades, the companies in America are going to end up.  Because you can’t buy creativity, you have to foster it. That means doing things that may not make sense in the short term, but will have big payoffs in the long run.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s