It isn’t easy being a software developer. This particular career is intellectually engaging, often never the same, and opens up whole new worlds if you consider the industries of gaming, social, and mobile. I’ll discuss some of the complexities of software development and engineering, as I’ve already discussed what it is and why you should study it.
Why is it difficult?
Software engineering is a diverse field of sub fields where every detail matters. Minor things like famous off by one errors can mean devastation for your software. The field is deep and the risks are diverse. Whether you’re designing software for rockets or simply for the next big app, the risks can be huge. I’ll talk about the general complexities of an Internet application, as there are so many of them and that’s primarily what I’ve worked on.
Front-end engineering is the field that encompasses most iPhone, Android, and current web applications. Engineers from this field control the UI, the navigation, and basically everything the user sees. They create innovative user interfaces, manage when they should request data from the server and how much of it, and have to deal with hellish problems like supporting multiple platforms that their customers unfortunately continue to use. The most difficult times I’ve had as a front-end engineer was generally making things pixel perfect and scaleable with data (I.e. new data won’t destroy the layout), managing multiple languages (oh Arabic…), rendering performance and elasticity (frames per second is an intimate metric), and testing across multiple platforms. I’ve also had a lot of fun bringing ideas to life and the joy of multi touch and responsive interfaces.
For the past 3+ years I’ve been laser focused on back-end engineering. Back-end engineering is the wizard behind the curtain of an application. The front-end needs to save data somewhere, and that is usually the responsibility of the back-end. Engineers from this field determine where you’ll save data, how you’ll save data, how to create a usable URL/API for front-end engineers to use, and optimizing on performance. There are several layers of back end engineering, whether it be platform, applications, devops, or analytics, there is a lot of room to grow in this space and a lot of things to do. The most difficult things I’ve experienced in this space are understanding current data storage incumbents (MySQL, Redis, PostgresSQL), learning about data structures and algorithms and more interestingly probabilistic data structures, dealing with concurrency and distributed systems, and finally managing thousands of machines. Once your data grows, you need an easy way to be in control of everything, which is where the Hadoop family becomes a natural progression. There’s always something new to learn and you are generally in control of the entire stack. However, moving between jobs is not as easy as every company might have their own way of doing things, so it’s something to consider.
There is a growing trend for developer lead operations engineering. I’ve set up an elastic cluster in AWS for my previous role. This is essentially dev ops. Imagine that a million users suddenly come to your internet enabled application tomorrow. What happens to your system? It probably will go down. However, in the world of AWS and elastic computing, imagine that you can keep adding computing power as you need it. Imagine that for every 10,000 users who came to your site, you added another 32-core machine, all while you were at home in bed. This is entirely possible with dev ops and is something I’ve tested and used in elastic load balancing. In addition, how do you handle fail-over? What happens if your data center, where the data is physically stored, goes offline. Having automatic fail over to another data center is also another role of DevOps / Operations. Manual intervention would be more operations, and I’ll discuss that next.
Operations engineering generally doesn’t involve coding as apparently many people don’t know how to code. I don’t buy that explanation but regardless, there is a distinction behind operations and developer operations (DevOps). Anyways, if there is a manual intervention required for failover or that you need to setup DNS settings etc., operations will take on this task for you.
Understanding your data is extremely critical. At some point, every maturing company will want to record as much information as they can and build a data warehouse and attempt to mine it (query it) to gain insights on how their product is doing. Every sane product manager will want to know how their product is performing, whether it is statistically better or worse, and whether that change is sustained and not accidental. At the heart of this field is determining what to measure, how to measure it, and where to store it and you can imagine how that gets complicated when you easily have over 1000 terabytes of information.
I won’t get into much detail on this but routing networks, determining if networks require upgrades (operations-related), if they are saturated, and balancing load across racks and being rack aware are network problems in my opinion. It’s easy to ignore the hardware for most of the disciplines up to this point (even the data center one), but for network engineering, you need to know that there is significant risk if all your backups are on the same rack of computers.
I haven’t had to do hardware engineering since high school as I avoided all hardware subjects since then. This level of engineering requires coding in a confined memory and CPU requirements and testing on embedded systems with varying CPU and OS technologies. I’d be interested to know what they use these days, but as it isn’t my speciality, I wouldn’t be able to comment on it.
After all of these varying complexities of software, there is still one very critical aspect: security. Security isn’t just about cryptography, it’s also about threat models and privacy concerns. At the end of the day, your code, your network, your intellectual property has to be secure. There’s varying concerns of security within the web browser, and similarly growing concerns on mobile devices (I.e. distribution of signed APKs or iOS images). This whole area of engineering requires it’s own focused study and is more or less important depending on your field. A basic understanding of it is critical but the mastery of it is more essential than the mastery of any other field as forgetting something could lead to the infiltration, leak, or embarrassment of your company. Any proper threat model can help reveal these for your organization and any organization that uses computers needs one.
I’ve outlined a number of sub disciplines within software engineering and what makes them complicated. Unfortunately I couldn’t give a one definition for why software engineering alone is difficult, but the culmination of all these topics brings enormous complexity to the field as a whole. I haven’t even touched upon the other elements of software engineering such as the mathematics or the management of projects, both of which deserve their own discussions. If there is one big takeaway, it’s that whatever sub discipline you want to learn, there is a lot of depth within that discipline and that the discipline is just a small piece of a large ecosystem.