Technically, that is.
The past three days were a whirlwind of mistake after mistake. It was like a series of unfortunate events. We were tired and we were sleepy, but we had to keep our head in the game.
And to be honest, I don’t even know how to describe it all. But somehow, it filled three pages of my journal each day, which summed up to 9 pages by Tuesday evening.
As part of my job, there are days when we need to work on the weekends for implementations, attending to issues, and tight deliverable end dates. My colleague and I swapped schedules last weekend and as the primary standby, I had to do the implementation of a project which we were only informed last Wednesday.
Now this implementation isn’t new to me as I spent my first year in my work doing it with my lead and when she resigned, everyone expected that if there’s someone who knows the process by heart next to my lead, it was me. Yes, I was the apprentice to the great MFP expert.
Upon knowing of the implementation, the team was worried that no “dress rehearsal” nor deployment steps were given. This was a new project and I’ve never had a hands-on experience on a “new project” before, but somehow I knew the process. We tried doing a “dress rehearsal” last Thursday and Friday but the files that were given were of no use. It fell rightly on the meme statement, “I never test my codes, but when I do, I do it in production”.
The implementation started at 3PM of Sunday and I was telling my colleague that we can probably go home by 11PM. When the checks and files were given by the other team, I was surprised that I had to do two activities while I was expecting for only one. My current lead clarified the effort with the business and I had no choice but to complete it. 11PM was moved to 1AM.
While I was doing the implementation and running the process in a scheduler, I was also monitoring the space in the back-end. And while I was moving files to other mount-points, I did something horrible that would taint and haunt me forever, there was not enough space in the system and I had an idea that everything was messed up. One chain was gone, and the other was corrupted. We tried making a new one for the other since there was no data yet, but to no avail. Our counterpart had to step in and try some fix binaries, but everything didn’t work. I never thought this would be a problem until we finally decided to just opt for a restore of the whole thing and contact the storage team for the back-up. We didn’t want to try things on production and the error we were getting were new to us.
The business POC was disappointed, but we had no choice. It was the back-out plan. The restore would usually take 5hours, but the storage team managed to restore the production in an hour, the problem was, one chain was still missing and we were back to our very first problem. We tried fixing it again and again and again until we decided to just let the seniors decide what to do.
A call was set up and they decided to take a backup copy before the Saturday batch ran, so we had to go back and run the Saturday batch again and from there, we can start in step 1. By this time, everyone was involved, especially the team who developed the new scripts and processes. And by this time, everyone was asking how this was tested and how it passed the UAT.
One person also asked the senior manager that what if the storage team could not find a back up version with the chain, he seriously answered that we pray then.
So it was decided, we were gonna re-do the whole process, with the correct scripts and correct paths. I went home at 12NN, Monday to sleep for five hours and take a shower. I went back to the office at 8PM to continue the work. This time, everything was set up – the Saturday batch had completed and all we needed to do was pray hard that everything will go well and we will be able to hand over production to the business the next day.
We pulled another all-nighter and when we were done, our food delivery came and we started eating but my appetite disappeared the instant I saw that production had no data for actuals in the elapsed period! My blood drained and my hands started shaking. I didn’t want to repeat the whole process. We were delayed! Victor, my colleague, who was with me the whole time assisting in everything he could get his hands on, stared at me and closed his eyes in defeat.
We were quiet for a minute and I started praying on my desk and I heard Victor called out to the Lord as well – I’m not sure if that was just his expression but nonetheless, he was well aware that God’s the only One who can help us at this time.
I started checking the log files and I was pretty sure that the data was loaded. I didn’t know what prompted me to check the maces and there I saw the logs which told us that the maces did not run due to the hard-coded paths. I felt like skinning myself then and there. The good thing though was that we didn’t have to re-do the whole thing. We had to manually run the maces to copy the values from one measure to another. The whole automation process was useless at that moment. I asked Victor to run the maces manually as he has the keenest eyes when editing values and at that time, we could not afford to mess up anything anymore. It was 3AM for crying out loud! Gladly, the maces ran for a good amount of only 20 minutes.
The business POC didn’t give a damn with the reconciliation, but instead suggested that we sum up the data that was given by the other team through their files and then compare it to production data.
Fortunately, everything was equal and we couldn’t be happier. Final checks were completed and the last step was executed which was to release the Monday batch. We took in everything and congratulated ourselves.
We went home at 9AM, Tuesday and I slept for five hours straight. When I woke up at around 1PM, I started writing on my journal again and meditated on how God has kept us strong and active all through the night.
I have prayed to the Lord for guidance and alertness before the activity started and I was somehow depressed as to how everything turned out. But when I came home that Monday morning and prayed more, I learned to accept the situation and asked God, that instead of changing it, I wanted to know what He wanted me to learn, what He wanted me to see, what He wanted me to understand.
On top of my list would be, God is in control. Even if the situation gets out of hand, I should always remember that I am leaning on the God of the Already! I should just do my thing in the best of my capability and I should know that He is also working in His own time. It’s okay that I test the waters, that I keep myself from drowning, because then He would teach me how to swim, and if not, He would just lift me out of it. And that’s what He did. When we were losing our sanity at 3AM, He gave us hope.
Second would be, if He doesn’t work on you, He works on someone else. In this activity, I believe He worked on Victor. I found Victor a blessing and someone I have leaned on. I had confidence in him and in his skills. I had a feeling that with him around, we had a way out. I really appreciated how he chose to stick into the situation like that and never left me alone to face everything. I will not also forget the management and the business POC. Though we messed up, they were there to guide us all the way without any pressure. I know that they also had to answer to men with higher positions, but I really appreciate that everything we dreaded didn’t happen.
Thirdly, I realized that the smallest mistake might cause or cost the company big. For my experience, we probably cost the business a day’s salary of 300 people who were not able to work on the work that they do. My stage was the production, almost the same as the hospital as the doctors and nurses’ stage; the counter as the bankers and the cashiers stage; the court as the athletes stage. Everything was critical, and one simple mistake can cost you a lot. The experience taught me that my job is important as much as yours, irregardless of the salary difference. Yet at the same time, it taught me that at the end of the day, this is just a job – something that does not lessen nor increases my worth.
Lastly, this shattered and heightened my faith. Psalms 77 would reflect my agony last Monday – it talked about how a human was calling onto God and how God was not answering to him, but then he realized that no matter what, God is good, and He will never change. He has been good before, now and after. It was just a matter of seeing God and seeing the situation. If we see and know God for who He really is, which is all powerful and almighty, we see the situation as something positive. But if we are not seeing God the way He really is, we will get depressed, hopeless and will probably pray to be not in the same situation ever again because it’s bad and nothing good will come out of it. My faith has moved another level, and God just showed me that He can move mountains.
So whatever mountain you have in your life right now, hand it up to the Lord and let Him do what He does best. Keep calm for our God is able!
Now to him who is able to do immeasurably more than all we ask or imagine, according to his power that is at work within us, to him be glory in the church and in Christ Jesus throughout all generations forever and ever. – Ephesians 3:20-21