02. Software Engineering Design & Security Principles

This is the continuation of the course Secure Coding in Software Engineering. Design Problems Wicked Problem[1] A problem that can be clearly defined only by solving it or solving a part of it (Horst Rittel and Melvin Webber, 1973). You essentially solve the problem first to define the problem, then solve it again to find a solution that works. "Design" is a wicked problem. Source: https://xkcd.com/2021/ Software Design Book Recommendation Software Engineering Design by Carlos Otero 2012 https://a.co/d/0cXzdeok Some concepts of software designs can be described as Abstraction: Deals with creation of conceptual entities that facilitates solving the problem by focusing on the essential characteristics of the entities themselves (procedural and data abstraction). Required for good modularization (Liskov and Guttag 2010). Modularization: Decomposition of the system until fine-grained components are created. Encapsulation: Principles that deals with providing access to the services of the conceptual entities (modules, components etc.) by exposing only essential information and hiding details on how those services are carried out. Coupling: The manner and degree of interdependence between software modules. Cohesion: The manner and degree to which the tasks performed by a single software module are related to one another. Separation of Interface and Implementation: While encapsulation deals with exposing and hiding implementation details, this principle of separation goes further and allows for an interface with the implementation (separate) to be swapped for modified or new behavior. Completeness: Measures how well design units provide required services to achieve their intent. Sufficiency: Measures how well design units provide services that are sufficient to achieve their intent. Impact of Design Principles on Security Abstraction and modularization help in being able to determine which of the modules are security relevant. If you don’t need a module, then it is redundant and should be removed. For example, having a module to deal with users and their profile management can allow for security related functions for a user to be implemented easily. Encapsulation and Interface-Implementation Separation can be used to achieve security goals for modules as well by checking security related functions as part of the design. This design principle also helps in reducing the number of surfaces that need to be checked or secured by separating the implementation from the interface. For example, exposing administrative functionality for a web based application to authenticated and authorized users only where the authentication and authorization mechanisms are implemented independently of the Login. Reducing coupling is a general software design goal and helps in securing modules because it also reduces the cascade of failures through the system. For Example, NPM repository down: March 2016, a disgruntled developer unpublished a very popular package called left-pad due to a naming dispute that many other packages had as a dependency causing widespread disruption[2]. Data abstraction is a key principle that has implications on privacy and securing data flows within an application. The sufficiency principle similarly has implications on minimizing the surfaces that need to be secured. Source: https://xkcd.com/2347/ NIST Special Publication 800-160, Volume 1 Revision 1 The following topics are taken from NIST Special Publication 800-160, Volume 1 Revision 1[3], Engineering Trustworthy Secure Systems: How does software quality affect security? There are several software quality metrics that can help assess the quality of the software and its design. High software quality can also lead to more secure software since it helps with the maintenance of software systems during its lifecycle. Source: https://xkcd.com/1513/ For example, In a code base, a developer may state that a method/function is not going to be used by anyone and can stay in the code base. Even if no other components depend on it, it is more than likely: That someone will reuse the code from the component That someone will make it dependent on another software module That an attacker will gain access to it Finding bugs becomes easier when software quality is maintained! Source: https://xkcd.com/1926/ Security Principles Weakness Mistakes or flaws (intentional or unintentional) in the software system that can lead to serious system level impacts. You can use the CWETM language to describe such flaws. Different from vulnerabilities, which involve using a weakness or set of weaknesses to achieve a desired effect or impact on the system. Attack Surface Informal Definition We have heard the concept of a "surface" in principles of software system design – an interface to the interaction between modules. An attack surface similarly provides attackers an in

Mar 9, 2025 - 10:35

02. Software Engineering Design & Security Principles

This is the continuation of the course Secure Coding in Software Engineering.

Design Problems

Wicked Problem^[1]

A problem that can be clearly defined only by solving it or solving a part of it (Horst Rittel and Melvin Webber, 1973). You essentially solve the problem first to define the problem, then solve it again to find a solution that works.

"Design" is a wicked problem.

Source: https://xkcd.com/2021/

Software Design

Book Recommendation

Software Engineering Design by Carlos Otero 2012 https://a.co/d/0cXzdeok

Some concepts of software designs can be described as

Abstraction: Deals with creation of conceptual entities that facilitates solving the problem by focusing on the essential characteristics of the entities themselves (procedural and data abstraction). Required for good modularization (Liskov and Guttag 2010).
Modularization: Decomposition of the system until fine-grained components are created.
Encapsulation: Principles that deals with providing access to the services of the conceptual entities (modules, components etc.) by exposing only essential information and hiding details on how those services are carried out.
Coupling: The manner and degree of interdependence between software modules.
Cohesion: The manner and degree to which the tasks performed by a single software module are related to one another.
Separation of Interface and Implementation: While encapsulation deals with exposing and hiding implementation details, this principle of separation goes further and allows for an interface with the implementation (separate) to be swapped for modified or new behavior.
Completeness: Measures how well design units provide required services to achieve their intent.
Sufficiency: Measures how well design units provide services that are sufficient to achieve their intent.

Impact of Design Principles on Security

Abstraction and modularization help in being able to determine which of the modules are security relevant. If you don’t need a module, then it is redundant and should be removed.
For example, having a module to deal with users and their profile management can allow for security related functions for a user to be implemented easily.
Encapsulation and Interface-Implementation Separation can be used to achieve security goals for modules as well by checking security related functions as part of the design. This design principle also helps in reducing the number of surfaces that need to be checked or secured by separating the implementation from the interface.
For example, exposing administrative functionality for a web based application to authenticated and authorized users only where the authentication and authorization mechanisms are implemented independently of the Login.
Reducing coupling is a general software design goal and helps in securing modules because it also reduces the cascade of failures through the system.
For Example, NPM repository down: March 2016, a disgruntled developer unpublished a very popular package called left-pad due to a naming dispute that many other packages had as a dependency causing widespread disruption^[2].
Data abstraction is a key principle that has implications on privacy and securing data flows within an application.
The sufficiency principle similarly has implications on minimizing the surfaces that need to be secured.

Source: https://xkcd.com/2347/

NIST Special Publication 800-160, Volume 1 Revision 1

The following topics are taken from NIST Special Publication 800-160, Volume 1 Revision 1^[3],
Engineering Trustworthy Secure Systems:

How does software quality affect security?

There are several software quality metrics that can help assess the quality of the software and its design. High software quality can also lead to more secure software since it helps with the maintenance of software systems during its lifecycle.

Source: https://xkcd.com/1513/

For example, In a code base, a developer may state that a method/function is not going to be used by anyone and can stay in the code base. Even if no other components depend on it, it is more than likely:

That someone will reuse the code from the component
That someone will make it dependent on another software module
That an attacker will gain access to it

Finding bugs becomes easier when software quality is maintained!

Source: https://xkcd.com/1926/

Security Principles

Weakness

Mistakes or flaws (intentional or unintentional) in the software system that can lead to serious system level impacts.
You can use the CWE^TM language to describe such flaws.
Different from vulnerabilities, which involve using a weakness or set of weaknesses to achieve a desired effect or impact on the system.

Attack Surface

Informal Definition

We have heard the concept of a "surface" in principles of software system design – an interface to the interaction between modules.
An attack surface similarly provides attackers an interface to system resources that could be abused to access security vulnerabilities and to exploit the software system to achieve an attacker’s goals.

For a home, what are some example attack surfaces?

Formal Definition

Attacker can attack using channels (e.g., ports, sockets), invoke methods (e.g., API), and send data items (input strings or indirectly via persistent or stored data)
A system’s attack surface – Subset of the system’s resources (channels, methods, and data) that can be used in attacks on the system.
More attack surfaces likely means it is easier to exploit and cause more damage.

Source: Attack Surface Metric, Pratyusa K. Manadhata, CMU-CS-08-152, November 2008

Attack Vector

The set of inputs, actions and effects on the software system that when provided via the attack surface cause the exploitation of the system achieving the attacker/user’s goal.
Typically involves:
- Access to the software system
- Ability to manipulate input
- Ability and access to the configuration of the software environment

Illustration

Death Star, Star Wars

Source: Johnson, S. (1995). In Star Wars technical journal. Boxtree.

The reactor system at the core has a weakness that when it is hit with a blast, it causes a chain reaction that can be catastrophic. Intentionally hidden by Galen Erso, a scientist coerced to work for the Galactic Empire.
Exhaust port on the Death Star designed for exhaust (output) is an example of an attack surface.
The Death Star has a vulnerability where anything that can cause a chain reaction in the reactor core can lead to a catastrophic loss of the system.
The attack vector is the two proton torpedoes used to access the exhaust port, reach the reactor core and initiate the chain reaction.

Typical Security Objectives

Confidentiality: Information is only available to those who should have access.
Integrity: Data is known to be correct and trusted.
Availability: Information is available for use by legitimate users when it is needed.

Source: https://www.itgovernance.co.uk/blog/what-is-the-cia-triad-and-why-is-it-important

Other Security Objectives

Non-repudiation - Transactional security property where sender of a message is provided with proof of delivery and recipient of the message is provided with proof of sender’s identity. A real-life example of non-repudiation is when you sign for a package upon delivery—your signature serves as proof that you received it, preventing you from later denying acceptance.
Privacy - Property where all disclosure of information is provided only by authorized consent. A real-life example of privacy is when a doctor keeps your medical records confidential, ensuring that only authorized personnel can access them.
Audit/Accountability/Logging - Transactional security property where all security relevant actions and events are recorded and can be examined by authorized parties. A real-life example of audit/accountability/logging is security cameras in a retail store recording all transactions, allowing store managers to review footage if a theft or dispute occurs.
Id-entity - Typically digital identity, a security property that allows unique identification of individual entities allowing for authentication, authorization and access to services. A real-life example of identity is a passport, which serves as official proof of a person's identity and nationality when traveling internationally.

Set of Security Principles

A set of guiding principles for securing software systems need to be followed. The real "secret-sauce" behind most of the secure designs of software being followed in modern enterprises.

The following security principles already have existed since the 1970s and are from the following sources:

McGraw, Gary & Viega, John. "Keep It Simple." Software Development. CMP Media LLC, May, 2003.
Viega, John & McGraw, Gary. Building Secure Software: How to Avoid Security Problems the Right Way. Boston, MA: Addison-Wesley, 2002.
Saltzer, Jerome H. & Schroeder, Michael D. "The Protection of Information in Computer Systems" 1278-1308. Proceedings of the IEEE 63, 9 (September 1975).
Howard, Michael & LeBlanc, David. Writing Secure Code. 2nd. Redmond, WA: Microsoft Press, 2002.

Trustworthiness vs. Trust^[4]

"Trustworthiness implies that something is worthy of being trusted."
"Trust merely implies that you trust something, whether it is trustworthy or not." - Trust is a decision. - You should only trust things that have adequate evidence of being trustworthy.

Security Principle - Defence in Depth

Do not rely on a single security method.
Anticipate failures and layer basic security practices.
- Example: A Bank that manages money.
  - Security Guard
    Title: A private security officer at a Chinese factory in February 2004. Creator: Robbie Sproule. Source: Flickr. License: CC-By-2.0
  - Bank Building
    Title: Lloyds Bank Building. Creator: Pit-Yacker. Source: Wikimedia Commons. License: CC-By-2.5
  - Bank Teller
    Title: A teller in a branch of Bank Muamalat. Creator: Melwinsy. Source: Wikipedia. License: CC-By-SA-4.0
  - Bank Vault Title: Bank Vaults under Hotels in Toronto, Ontario. Creator: Jason Baker. Source: Flickr. License: CC-By-2.0
- The above example demonstrates that to get to the bank vault, one must pass through multiple steps of security to reach it.

Security Principle - Keep Security Simple

Keep security or protection mechanisms simple.
Makes it easy to understand, implement and analyze security.
- This has a direct co-relation to a CWE called CWE-655: Insufficient Psychological Acceptability.
Can also make it easy to use.

Security Principle - Attack Surface Reduction

Actively address and reduce any surfaces exposed to unauthenticated or unauthorized actors that allow for the system to be compromised against its security goals.
This can be done in a number of ways:
- From software design principles, the ideas of encapsulation or hiding functionality from entities (code, users, etc.) that don’t require it, separation of interface and implementation can also be used to control which entities have access to what functionality etc.
- In software systems, easiest way is to follow:
  - KISS principle: 'keep it simple stupid'. Keeping the system simple at design time tends to make it easy for actors to use the system and prevent misuse. It also helps in understanding and analyzing the system for attack surfaces and vulnerabilities.
  - Remove functionality that is not needed. The attack surface is essentially going to be all the code, functionality or data that is accessible by entities. Removing or reducing those can go a long way. Example could be keeping a code that we feel nobody is going to use, in the codebase which eventually someone ends up using bringing up weaknesses.

Security Principle - Establish Secure Defaults

The default configuration or state that ships with the software system should already include the most secure configuration. That means out of the box the software system should stop insecure actions.
Do not rely on an entity to configure the software system in order to enable security features or to make it run securely.
Example: A banking app requires strong passwords and enables two-factor authentication (2FA) by default, ensuring users have a secure account setup from the start.

Security Principle - Principle of Least Privilege

Not everyone needs access or privileges to do everything.
The actors (users, or programs) that use the software system should only be allowed to operate with the least privileges possible.
This limits the damage from an accident, error or attack.
Example: A hotel key card that only grants access to a guest’s room and common areas, but not to staff-only areas or other guests' rooms.

Security Principle - Fail Securely

Security controls in an application should assume the application is under attack by default.

Apple SSLVerifySignedServerKeyExchange checks validity of SSL certificate:

The second goto fail is not part of the if block:

if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
    goto fail;
    goto fail;
... other checks ...

fail:
    ... buffer frees (cleanups) ...
    return err;

Source: The Apple goto fail vulnerability: lessons learned, David A. Wheeler

Several issues here. But, one of them is the error code default was 0 which meant no error. This allowed attacker to skip other checks and return error of 0 as long as first statement returned no error. Here, failure cases should be checked explicitly and independently rather than stacking them one after another.^[5]

Security Principle - Complete Mediation ("Non-Passable")

Following from the earlier example, this principle involves making sure:
- To find all channels
- Check all inputs from untrusted sources
- Check as soon as possible and fail early
If the system has a client-server model, this means ensuring:
- Security checks are done on the server-side (client is not necessarily trusted, server may be trusted but not necessary)
- Check trustworthiness of environment (client or server) to ensure checks can also return trustworthy results
- Client-side checking can improve user-response and lower server load, however do client-side checks in addition to server-side checks.
Example: A credit card transaction where the bank verifies the card details and available balance every time a purchase is made, rather than relying on previously approved transactions. This ensures that all inputs are checked, trustworthiness is validated on the bank's secure server, and potential fraud is caught early.

Security Principle - Don't Trust Service Interface

This principle means the software system should not trust any of the interfaces it interacts with especially with regard to:
- Its behavior
- Its promises
The mere existence of an interface is not a contract that it will behave correctly.
Example interfaces:
- Third-party services and libraries
- Databases
- File systems
- Memory

Source: [6]

Security Principle - Separation of Duties/Privilege

This principle involves splitting the roles, duties and privileges that go along with those roles and duties into separate parts of the software system.
Example: You would not want a bank to allow the same customer to change the address of an account and authorize a check against an account.

Security Principle - Avoid Security Through Obscurity

This principle states:
- Avoid implementing obscure security mechanisms to "add" security.
- Adversaries and attackers likely already have access to the software system, source code and perhaps even the data for reverse engineering given enough resources.
- Relying on such mechanisms is very dangerous!
Assume the software system is already compromised and build security in.
Example: The gate here really does not really add any security nor does it add deterrence.

Security Principle - Avoid Non-Atomic Security Operations

This principles talks to how the software system should treat and implement security actions and operations atomically.
Example:
- Fake check scheme of the past.
- Both check verification and account balance update should be part of the same atomic transaction.
- Check needs to be verified and then the deposit updated as part of the same transaction.

Security Principle - Fix Security Issues Correctly

Security issues need to be treated at the root cause of the issue rather than just addressing symptoms.
The first gut response to an attack is to fix the issues responsible directly for the attack. You may only be addressing the symptoms of the problem.
However, this principle states that the software system and processes should be analyzed for a deeper analysis on what allowed the issue to happen in the first place.
As a software engineer you need to be able to fully understand the problem before you can design and implement a fix.

Security Principle - Least Common 'Security' Mechanism

This principle is actually a design principle where the system should be implemented in such a way as to reduce and minimize the use of shared mechanisms. For example, the use of /tmp or /var/tmp directories to store temporary files for users.
This applies to security mechanisms as well.
Shared objects provide potentially dangerous channels for information flow and unintended interactions.

What is wrong with this code?

public class ApplicationUser : IdentityUser<int, ApplicationUserLogin,ApplicationUserRole, ApplicationUserClaim>, User<int>
{
    public string Name { get; set; }
    public string Surname { get; set; }
    //code omitted for brevity
}

public class Student: ApplicationUser
{
    public int? Number { get; set; }

    [HttpPost]
    [ValidateAntiForgeryToken]
    public JsonResult Update([Bind(Exclude = null)] StudentViewModel model)
    {
        if (ModelState.IsValid)
        {
            ApplicationUser user = UserManager.FindById(model.Id);
            user = new Student
            {
                Name = model.Name,
                Surname = model.Surname,
                UserName = model.UserName,
                Email = model.Email,
                PhoneNumber = model.PhoneNumber,
                Number = model.Number, //custom property
                PasswordHash = checkUser.PasswordHash
            };
            UserManager.Update(user);
        }
    }
}

Source: [7]

Security Principle - Limit Resource Dependencies

This principle involves limiting the amount of resources depended upon by the software system wherever possible.
Fail2Ban – Limits excessive unauthorized login attempts over the network for an application based on logs and bans them when the limits are exceeded. Can be used for multiple applications including SSH.
Google, 2020 – Denial Of Service by UDP amplification attack^[8]:
- "In 2017, our Security Reliability Engineering team measured a record-breaking UDP amplification attack sourced out of several Chinese ISPs (ASNs 4134, 4837, 58453, and 9394), which remains the largest bandwidth attack of which we are aware."
- "The attacker used several networks to spoof 167 Mpps (millions of packets per second) to 180,000 exposed CLDAP, DNS, and SMTP servers, which would then send large responses to us. This demonstrates the volumes a well-resourced attacker can achieve. This was four times larger than the record-breaking 623 Gbps attack from the Mirai botnet a year earlier." - Damien Menscher, Security Reliability Engineer at Google

Security Principle - Ease Of Use For The Software System

This principle follows the previously discussed KISS principle.

However, as applied to security, the software system's human interface must be designed for ease of use so users will routinely and automatically use the protection mechanisms correctly.
Making it easy for users to secure their data and the software system, will make the system more secure.
Mistakes are reduced when security mechanisms in the software system closely match the user’s perception of their own protection goals. - Credit: Dr. David A. Wheeler

Security Principle - Hardening The Software System Environment

The software system is designed to run in an environment.

Building an application securely + deploying it in an insecure environment ≠ Secure Software System
The software system should also not assume anything about the environment wherever possible.

Types of Security Problems

We will discuss some type of security problems that are worth mentioning outside of the security principles.

The "Confused" deputy problem

A more privileged program ("a deputy") that is tricked by a lower privileged program into transitively giving it authority and misusing the authority. This is a type of privilege escalation.

Confused Deputy Example (Mainframe Billing System)

In an old mainframe billing system, a privileged billing process is responsible for generating and finalizing invoices. However, due to weak access controls, a user (or another less-privileged process) can trick the billing system into overwriting a bill—either by abusing its elevated permissions or by injecting unauthorized inputs.

Why is this a Confused Deputy Problem?

The billing system (deputy) has the authority to modify bills.
A user or process (attacker) without direct billing privileges tricks the system into writing incorrect values.
The billing system, acting on behalf of the user, unknowingly misuses its higher privilege, allowing unauthorized bill modifications.

A more modern analogy is Cloud Storage Misuse—where a web application (acting as a deputy) has write access to a cloud storage bucket. If a user uploads a malicious file, and the app inadvertently writes it with admin privileges, it allows unauthorized file modifications, similar to the mainframe billing issue.

Time-of-check-Time-of-Use (TOCTOU) problem

A race condition where a program checks the state of a part of the system (like a security credential) and the use of the results of that check.

Example: The setuid program bug^[9]. The following code had the race condition:
```
if (access("file", W_OK) != 0) {
    exit(1);
}

fd = open("file", O_WRONLY);
write(fd, buffer, sizeof(buffer));
```
Where, After the access check, before the open, the attacker replaces file with a symlink to the Unix password file /etc/passwd:
```
symlink("/etc/passwd", "file");
```

Covert/Side Channels

Surreptitious ways to send or "leak" data across data boundaries.
This requires some sort of access, coordination and/or sharing.

Impacts from Security Principle Violations

Information leakage: Sensitive or Unauthorized information got exposed.
Modification of data: Sensitive data was modified or unauthorized changes made to data.
Denial of Service:
- Unreliable Execution: The software system became unreliable or went into a degraded mode of operation.
- Resource Consumption: The software system ran out of resources.
Unauthorized code or command execution: Unauthorized code or commands were able to be run or executed by external users.
Escalation of privilege / Assumption of Identity: Malicious actors were able to hijack the identity of a privileged user or escalate their privileges within the software system or its environment.
Bypass Protection Mechanism: Any protection mechanisms were easily bypassed or circumvented.
Hide Activities: Activities performed on the software system or its environment were hidden from auditors or system evaluators or monitors.
Loss of trust: Users lost trust in the system they were using.

References

Wicked Problem - Wikipedia
npm left-pad incident
Ron Ross, Michael McEvilley, Maichael McEvilley. Engineering of Trustworthy Secure Systems. Vol 1 Rev 1. https://doi.org/10.6028/NIST.SP.800-160v1r1.
Definitions from "Principled Assuredly Trustworthy Composable Architectures" by Peter Neumann, 2004
Failing Securely
Is there any real-world “science” behind R2-D2’s computer interfacing arm?
How to save new record with hashed password in my custom table instead of aspnet user?
Five Most Famous DDoS Attacks and Then Some
Time-of-check to time-of-use