Operating System Security (131192)

After studying this course, you should be able to:

Lets get started! (copy)



An OS associates a set of privileges with each process. These privileges dictate what resources the process may access, including regions of memory, files, and privileged system instructions. Typically, a process that executes on behalf of a user has the privileges that the OS recognizes for that user. A system or utility process may have privileges assigned at configuration time.On a typical system, the highest level of privilege is referred to as administrator,supervisor, or root access. One root access provides access to all the functions and services of the operating system. With root access, a process has complete control of the system and can add or change programs and files, monitor other processes, send and receive network traffic, and alter privileges. 

A key security issue in the design of any OS is to prevent, or at least detect,attempts by a user or a piece of malicious software (malware) from gaining unauthorized privileges on the system and, in particular, from gaining root access. 

Intruders and Malicious Software (copy)

System Access Threats

System Access Threats

System access threats fall into two general categories:

  • Intruders

  • Malicious software



One of the most common threats to security is the intruder (the other is viruses), often referred to as a hacker or cracker. In an important early study of intrusion, Anderson identifies three classes of intruders:

  • Masquerader: An individual who is not authorized to use the computer and who penetrates a system’s access controls to exploit a legitimate user's account.
  • Misfeasor: A legitimate user who accesses data, programs, or resources for which such access is not authorized, or who is authorized for such access but misuses his or her privileges.
  • Clandestine user: An individual who seizes supervisory control of the system and uses this control to evade auditing and access controls or to suppress audit collection.

Intruder attacks range from the benign to the serious. At the benign end of the scale, there are many people who simply wish to explore internet and see what is out there. At the serious end are individuals who are attempting to read privileged data, perform unauthorized modifications to data, or disrupt the system.

The objective of the intruder is to gain access to a system or to increase the range of privileges accessible on a system. Most initial attacks use system or software vulnerabilities that allow a user to execute code that opens a backdoor into the system. Intruders can get access to a system by exploiting attacks such as buffer overflows on a program that runs with certain privileges. We introduce buffer overflow attacks later in this course. 

Alternatively, the intruder attempts to acquire information that should have been protected. In some cases, this information is in the form of a user password.With knowledge of some other user’s password, an intruder can log in to a system and exercise all the privileges accorded to the legitimate user.

Match the three classes of intruders with their personalitiy

  • Masquerader
    an outsider
  • Misfeasor
    an insider
  • Clandestine user
    either an outsider or an insider

Malicious software (Malware)

Perhaps the most sophisticated types of threats to computer systems are presented by programs that exploit vulnerabilities in computing systems.Such threats are referred to as malicious software, or malware. In this context, we are concerned with threats to application programs as well as utility programs, such as editors and compilers, and kernel-level programs.

Malicious software can be divided into two categories

  • those that need a host program and 
  • those that are independent.

 The former, referred to as parasitic, are essentially fragments of programs that cannot exist independently of some actual application program, utility, or system program. Viruses, logic bombs, and backdoors are examples. 

The latter are self-contained programs that can be scheduled and run by the operating system. Worms and bot programs are examples.

We can also differentiate between those software threats that:

  •  do not replicate and 
  • those that do

The former are programs or fragments of programs that are activated by a trigger. Examples are logic bombs, backdoors, and bot programs. 

The latter consists of either a program fragment or an independent program that, when executed, may produce one or more copies of itself to be activated later on the same system or some other system. Viruses and worms are examples.

Malicious software can be relatively harmless or may perform one or more of a number of harmful actions, including destroying files and data in main memory, bypassing controls to gain privileged access, and providing a means for intruders to bypass access controls.

What are the examples of malicious software who needs host program?

  • Viruses, logic bombs and backdoors
  • Worms and bot programs
  • Logic bombs, backdoors and bot programs
  • Viruses and worms



 RFC 4949 (Internet Security Glossary) defines intrusion detection as follows: 

A security service that monitors and analyzes system events for the purpose of finding, and providing real-time or near real-time warning of,attempts to access system resources in an unauthorized manner.

Intrusion detection systems (IDSs) can be classified as follows:

  •  Host-based IDS: Monitors the characteristics of a single host and the events occurring within that host for suspicious activity
  •  Network-based IDS: Monitors network traffic for particular network segments or devices and analyzes network, transport, and application protocols to identify suspicious activity.

An IDS comprises three logical components:

  • Sensors: Sensors are responsible for collecting data. The input for a sensor may be any part of a system that could contain evidence of an intrusion. Types of input to a sensor include network packets, log files, and system call traces.Sensors collect and forward this information to the analyzer.
  • Analyzers: Analyzers receive input from one or more sensors or from other analyzers. The analyzer is responsible for determining if an intrusion has occurred.The output of this component is an indication that an intrusion has occurred. The output may include evidence supporting the conclusion that an intrusion occurred. The analyzer may provide guidance about what actions to take as a result of the intrusion.
  •  User interface: The user interface to an IDS enables a user to view output from the system or control the behavior of the system. In some systems, the user interface may equate to a manager, director, or console component.Intrusion detection systems are typically designed to detect human intruder behavior as well as malicious software behavior.


 In most computer security contexts, user authentication is the fundamental building block and the primary line of defense. User authentication is the basis for most types of access control and for user accountability. RFC 4949 defines user authentication as follows: 

The process of verifying an identity claimed by or for a system entity. An authentication process consists of two steps:

  • Identification step: Presenting an identifier to the security system (Identifiers should be assigned carefully, because authenticated identities are the basis for other security services, such as access control service.)
  • Verification step: Presenting or generating authentication information that corroborates the binding between the entity and the identifier.

Example of Authentication

Four general means of authenticating a user’s identity

There are four general means of authenticating a user’s identity, which can be used alone or in combination:

  • Something the individual knows: Examples include a password, a personal identification number (PIN), or answers to a prearranged set of questions.
  • Something the individual possesses: Examples include electronic keycards, smart cards, and physical keys. This type of authenticator is referred to as a token.
  •  Something the individual is (static biometrics): Examples include recognition by fingerprint, retina, and face.
  •  Something the individual does (dynamic biometrics): Examples include recognition by voice pattern, handwriting characteristics, and typing rhythm.

All of these methods, properly implemented and used, can provide secure user authentication. However, each method has problems. An adversary may be able to guess or steal a password. Similarly, an adversary may be able to forge or steal a token. A user may forget a password or lose a token. Further, there is a significant administrative overhead for managing password and token information on systems and securing such information on systems. With respect to biometric authenticators,there are a variety of problems, including dealing with false positives and false negatives,user acceptance, cost, and convenience.

What is token?

  • Something the individual knows
  • Something the individual possesses
  • Something the individual is (static biometrics)
  • Something the individual does (dynamic biometrics)

Access control

Access control

Access control implements a security policy that specifies who or what (e.g., in the case of a process) may have access to each specific system resource and the type of access that is permitted in each instance.

An access control mechanism mediates between a user (or a process executing on behalf of a user) and system resources, such as applications, operating systems,firewalls, routers, files, and databases. The system must first authenticate a user seeking access. Typically, the authentication function determines whether the user is permitted to access the system at all. Then the access control function determines if the specific requested access by this user is permitted. A security administrator maintains an authorization database that specifies what type of access to which resources is allowed for this user. The access control function consults this database to determine whether to grant access. An auditing function monitors and keeps a record of user accesses to system resources.


Firewalls can be an effective means of protecting a local system or network of systems from network-based security threats while affording access to the outside world via wide area networks and the Internet. Traditionally, a firewall is a dedicated computer that interfaces with computers outside a network and has special security precautions built into it in order to protect sensitive files on computers within the network. It is used to service outside network, especially Internet connections and dial-in lines. Personal firewalls that are implemented in hardware or software, and associated with a single workstation or PC, are also common.

[BELL94] lists the following design goals for a firewall:

  1. The firewall acts as a choke point, so that all incoming traffic and all outgoing traffic must pass through the firewall. This is achieved by physically blocking all access to the local network except via the firewall.
  2. The firewall enforces the local security policy, which defines the traffic that is authorized to pass. Various types of firewalls are used, which implement various types of security policies, as explained later in this chapter.
  3. The firewall is secure against attacks. This implies the use of a hardened system with a secured operating system. Trusted computer systems are suitable for hosting a firewall and often required in government applications.

What is called network protection from network-based security threats?

  • Firewalls
  • Access control
  • Authentication

Buffer Overflow (copy)

Buffer Overflow Attacks

Main memory and virtual memory are system resources subject to security threats and for which security countermeasures need to be taken. The most obvious security requirement is the prevention of unauthorized access to the memory contents of processes. The security threats and countermeasures discussed in the preceding section are relevant to this type of memory protection.

Buffer overflow attack

Buffer overflow, also known as a buffer overrun:

 A condition at an interface under which more input can be placed into a buffer or data-holding area than the capacity allocated, overwriting other information. Attackers exploit such a condition to crash a system or toinsert specially crafted code that allows them to gain control of the system.

A buffer overflow can occur as a result of a programming error when a process attempts to store data beyond the limits of a fixed-sized buffer and consequently overwrites adjacent memory locations. These locations could hold other program variables or parameters or program control flow data such as return addresses and pointers to previous stack frames. The buffer could be located on the stack, in the heap, or in the data section of the process. The consequences of this error include corruption of data used by the program, unexpected transfer of control in the program,possibly memory access violations, and very likely eventual program termination.When done deliberately as part of an attack on a system, the transfer of control could be to code of the attacker’s choosing, resulting in the ability to execute arbitrary code with the privileges of the attacked process. Buffer overflow attacks are one of the most prevalent and dangerous types of security attacks.

To illustrate the basic operation of a common type of buffer overflow, known as stack overflow, consider the C main function given in Figure 15.1 a. This contains three variables (valid, str1, and str2), whose values will typically be saved in adjacent memory locations. Their order and location depends on the type of variable (local or global), the language and compiler used, and the target machine architecture. For this example, we assume that they are saved in consecutive memory locations, from highest to lowest, as shown in Figure 15.2. This is typically the case for local variables in a C function on common processor architectures such as the Intel Pentium family. The purpose of the code fragment is to call the function next_tag(str1) to copy into str1 some expected tag value. Let’s assume this will be the string START. It then reads the next line from the standard input for the program using the C library gets() function, and then compares the string read with the expected tag. If the next line did indeed contain just the string START, this comparison would succeed and the variable valid would be set to TRUE. This case is shown in the first of the three example program runs in Figure 15.1 b. Any other input tag would leave it with the value FALSE. Such a code fragment might be used to parse some structured network protocol interaction or formatted text file.

The problem with this code exists because the traditional C library gets()function does not include any checking on the amount of data copied. It reads the next line of text from the program’s standard input up until the first newline character occurs and copies it into the supplied buffer followed by the NULL terminator used with C strings. If more than seven characters are present on the input line, when read in they will (along with the terminating NULL character) require more room than is available in the str2 buffer. Consequently, the extra characters will overwrite the values of the adjacent variable, str1 in this case. For example, if the input line contained EVILINPUTVALUE, the result will be that str1 will be overwritten with the characters TVALUE, and str2 will use not only the eight characters allocated to it but seven more from str1 as well. This can be seen in the second example run in Figure 15.1b. The overflow has resulted in corruption of a variable not directly used to save the input. Because these strings are not equal, valid also retains the value FALSE. Further, if 16 or more characters were input, additional memory locations would be overwritten.

To exploit any type of buffer overflow, such as those we have illustrated here,the attacker needs:

  1. To identify a buffer overflow vulnerability in some program that can be triggered using externally sourced data under the attackers control.
  2. To understand how that buffer will be stored in the processes memory, and hence the potential for corrupting adjacent memory locations and potentially altering the flow of execution of the program.

Identifying vulnerable programs may be done by inspection of program source,tracing the execution of programs as they process over sized input, or using tools such as fuzzing, to automatically identify potentially vulnerable programs. What the attacker does with the resulting corruption of memory varies considerably, depending on what values are being overwritten.

Compile-Time Defenses

Finding and exploiting a stack buffer overflow is not that difficult. The large numberof exploits over the previous couple of decades clearly illustrates this. There is consequentlya need to defend systems against such attacks by either preventing themor at least detecting and aborting such attacks. Countermeasures can be broadlyclassified into two categories:

  • Compile-time defenses, which aim to harden programs to resist attacks
  • Runtime defenses, which aim to detect and abort attacks in executing programs programs

Compile-time defenses aim to prevent or detect buffer overflows by instrumenting programs when they are compiled. The possibilities for doing this range from choosing a high-level language that does not permit buffer overflows to encouraging safe coding standards, using safe standard libraries, or including additional code to detect corruption of the stack frame.

Choice of Programming Language and Safe Coding Techniques

Choice of Programming Language

One possibility is to write the program using a modern high-level programming language, one that has a strong notion of variable type and what constitutes permissible operations on them. Such languages are not vulnerable to buffer overflow attacks, because their compilers include additional code to enforce range checks automatically, removing the need for the programmer to explicitly code them. The flexibility and safety provided by these languages does come at a cost in resource use, both at compile time and also in additional code that must execute at run time to impose checks such as that on buffer limits.These disadvantages are much less significant than they used to be, due to the rapid increase in processor performance.

Safe Coding Techniques

In order to harden operating systems, the programmer needs to inspect the code and rewrite any unsafe coding constructs in a safe manner. Given the rapid uptake of buffer overflow exploits, this process has begun in some cases. A good example is the OpenBSD project, which produces a free, multiplatform 4.4BSD-based UNIX-like operating system. Among other technology changes, programmers have undertaken an extensive audit of the existing code base, including the operating system,standard libraries, and common utilities. This has resulted in what is widely regarded as one of the safest operating systems in widespread use.

Language Extensions and Use of Safe Libraries

Language Extensions and Use of Safe Libraries

Given the problems that can occur in C with unsafe array and pointer references, there have been a number of proposals to augment compilers to automatically insert range checks on such references. While this is fairly easy for statically allocated arrays, handling dynamically allocated memory is more problematic, because the size information is not available at compile-time. Handling this requires an extension to the semantics of a pointer to include bounds information and the use of library routines to ensure that these values are set correctly.  However, there is generally a performance penalty with the use of such techniques that may or may not be acceptable. These techniques also require all programs and libraries that require these safety features to be recompiled with the modified compiler. While this can be feasible for a new release of an operating system and its associated utilities, there will still likely be problems with third-party applications.

A common concern with C comes from the use of unsafe standard library routines, especially some of the string manipulation routines. One approach to improving the safety of systems has been to replace these with safer variants.Using these requires rewriting the source to conform to the new safer semantics. Alternatively, it involves replacement of the standard string library with a safer variant. Libsafe is a well-known example of this. It implements the standard semantics but includes additional checks to ensure that the copy operations do not extend beyond the local variable space in the stack frame. So, while it cannot prevent corruption of adjacent local variables, it can prevent any modification of the old stack frame and return address values, and thus prevent the classic stack buffer overflow types of attack we examined previously. This library is implemented as a dynamic library, arranged to load before the existing standard libraries, and can thus provide protection for existing programs without requiring them to be recompiled, provided they dynamically access the standard library routines(as most programs do). The modified library code has been found to typically be at least as efficient as the standard libraries, and thus its use is an easy way of protecting existing programs against some forms of buffer overflow attacks.

Stack Protection Mechanisms

Stack Protection Mechanisms

An effective method for protecting programs against classic stack overflow attacks is to instrument the function entry and exit code to set up and then check its stack frame for any evidence of corruption. If any modification is found, the program is aborted rather than allowing the attack to proceed.

Stackguard is one of the best-known protection mechanisms. It is a GCC(GNU Compiler Collection) compiler extension that inserts additional function entry and exit code. The added function entry code writes a canary value below the old frame pointer address, before the allocation of space for local variables. The added function exit code checks that the canary value has not changed before continuing with the usual function exit operations of restoring the old frame pointer and transferring control back to the return address. Any attempt at a classic stack buffer overflow would have to alter this value in order to change the old frame pointer and return addresses and would thus be detected, resulting in the program being aborted. For this defense to function successfully, it is critical that the canary value be unpredictable and should be different on different systems. If this were not the case, the attacker would simply ensure the shell code included the correct canary value in the required location. Typically, a random value is chosen as the canary value on process creation and saved as part of the processes state. The code added to the function entry and exit then uses this value.

Runtime Defenses

Runtime Defenses

Runtime Defenses provide some protection for existing vulnerable programs. These defenses involve changes to the memory management of the virtual address space of processes. These changes act either to alter the properties of regions of memory or to make predicting the location of targeted buffers sufficiently difficult to thwart many types of attacks.

Executable Address Space Protection

Many of the buffer overflow attacks involve copying machine code into the targeted buffer and then transferring execution to it. A possible defense is to block the execution of code on the stack,on the assumption that executable code should only be found elsewhere in the processes address space.

To support this feature efficiently requires support from the processor’s memory management unit (MMU) to tag pages of virtual memory as being nonexecutable.

Making the stack (and heap) nonexecutable provides a high degree of protection against many types of buffer overflow attacks for existing programs; hence the inclusion of this practice is standard in a number of recent operating systems releases. However, one issue is support for programs that do need to place executable code on the stack. Nonetheless, this is regarded as one of the best methods for protecting existing programs and hardening systems against some attacks.

Address Space Randomization and Guard Pages

Address Space Randomization

Another runtime technique that can be used to thwart attacks involves manipulation of the location of key data structures in the address space of a process. In particular, recall that in order to implement the classic stack overflow attack, the attacker needs to be able to predict the approximate location of the targeted buffer. The attacker uses this predicted address to determine a suitable return address to use in the attack to transfer control to the shellcode. One technique to greatly increase the difficulty of this prediction is to change the address at which the stack is located in a random manner for each process.

Another target of attack is the location of standard library routines. In an attempt to bypass protections such as nonexecutable stacks, some buffer overflow variants exploit existing code in standard libraries. These are typically loaded at the same address by the same program. To counter this form of attack, we can use a security extension that randomizes the order of loading standard libraries by a program and their virtual memory address locations. This makes the address of any specific function sufficiently unpredictable as to render the chance of a given attack correctly predicting its address very low.

Guard Pages

A final runtime technique that can be used places guard pages between critical regions of memory in a processes address space. Again, this exploits the fact that a process has much more virtual memory available than it typically needs. Gaps are placed between the ranges of addresses used for each of the components of the address space. These gaps, or guard pages, are flagged in the MMU as illegal addresses, and any attempt to access them results in the process being aborted. This can prevent buffer overflow attacks, typically of global data,which attempt to overwrite adjacent regions in the processes address space.

What are runtime defenses?

  • Countermeasures, which aim to harden programs to resist attacks.
  • Countermeasures, which aim to detect and abort attacks in executing programs programs.

Why Stackguard additional function is successfully?

  • Because canary value is unpredictable.
  • Because additional function inserts entry and exit code

Access Control (copy)

File System Access Control

Following successful logon, the user has been granted access to one or a set of hosts and applications. This is generally not sufficient for a system that includes sensitive data in its database. Through the user-access control procedure, a user can be identified to the system. Associated with each user, there can be a profile that specifies permissible operations and file accesses. The operating system can then enforce rules based on the user profile. The database management system, however, must control access to specific records or even portions of records. The issue is more than just a matter of level of detail. Whereas the operating system may grant a user permission to access a file or use an application, following which there are no further security checks, the database management system must make a decision on each individual access attempt. That decision will depend not only on the user’s identity but also on the specific parts of the data being accessed and even on the information already divulged to the user.

A general model of access control as exercised by a file or database management system is that of an access matrix. The basic elements of the model are as follows:

  •  Subject: An entity capable of accessing objects. Generally, the concept of subject equates with that of process. Any user or application actually gains access to an object by means of a process that represents that user or application.

  • Object: Anything to which access is controlled. Examples include files, portions of files, programs, segments of memory, and software objects (e.g., Java objects).
  • Access right: The way in which an object is accessed by a subject. Examples are read, write, execute, and functions in software objects.

An access matrix is usually sparse and is implemented by decomposition in one of two ways. The matrix may be decomposed by columns, yielding access control lists. Thus for each object, an access control list lists users and their permitted access rights. The access control list may contain a default,or public, entry. This allows users that are not explicitly listed as having special rights to have a default set of rights. Elements of the list may include individual users as well as groups of users.

Decomposition by rows yields capability tickets. A capability ticket specifies authorized objects and operations for a user. Each user has a number of tickets and may be authorized to loan or give them to others. Because tickets maybe dispersed around the system, they present a greater security problem than access control lists. In particular, the ticket must be unforgeable. One way to accomplish this is to have the operating system hold all tickets on behalf of users. These tickets would have to be held in a region of memory inaccessible to users.

Network considerations for data-oriented access control parallel those for user-oriented access control. If only certain users are permitted to access certain items of data, then encryption may be needed to protect those items during transmission to authorized users. Typically, data access control is decentralized, that is,controlled by host-based database management systems. If a network database server exists on a network, then data access control becomes a network function.

Access Control Policies

An access control policy dictates what types of access are permitted, under whatcircumstances, and by whom. Access control policies are generally grouped into thefollowing categories:

  • Discretionary access control (DAC): Controls access based on the identity of the requestor and on access rules (authorizations) stating what requestors are(or are not) allowed to do. This policy is termed discretionary because an entity might have access rights that permit the entity, by its own volition, to enable another entity to access some resource.
  •  Mandatory access control (MAC): Controls access based on comparing security labels (which indicate how sensitive or critical system resources are) with security clearances (which indicate system entities are eligible to access certain resources). This policy is termed mandatory because an entity that has clearance to access a resource may not, just by its own volition, enable another entity to access that resource.
  •  Role-based access control (RBAC): Controls access based on the roles that users have within the system and on rules stating what accesses are allowed to users in given roles.

These three policies are not mutually exclusive (Figure 15.4). An access control mechanism can employ two or even all three of these policies to cover different classes of system resources.

Discretionary Access Control

The model assumes a set of subjects, a set of objects, and a set of rules that govern the access of subjects to objects. We can identify three requirements: representing the protection state, enforcing access rights, and allowing subjects to alter the protection state in certain ways.

To represent the protection state, we extend the universe of objects in the access control matrix to include the following:

  • Processes: Access rights include the ability to delete a process, stop (block),and wake up a process.
  • Devices: Access rights include the ability to read/write the device, to control its operation (e.g., a disk seek), and to block/unblock the device for use.
  • Memory locations or regions: Access rights include the ability to read/write certain locations of regions of memory that are protected so that the default is that access is not allowed. 
  • Subjects: Access rights with respect to a subject have to do with the ability to grant or delete access rights of that subject to other objects, as explained subsequently.

Figure 15.6 suggests that every access by a subject to an object is mediated by the controller for that object, and that the controller’s decision is based on the current contents of the matrix. In addition, certain subjects have the authority to make specific changes to the access matrix. A request to modify the access matrix is treated as an access to the matrix, with the individual entries in the matrix treated as objects. Such accesses are mediated by an access matrix controller, which controls updates to the matrix.

The model also includes a set of rules that govern modifications to the access matrix, shown in Table 15.1. For this purpose, we introduce the access rights owner and control and the concept of a copy flag, explained in the subsequent paragraphs.

The first three rules deal with transferring, granting, and deleting access rights.Suppose that the entry a* exists in A[S0, X ]. This means that S0 has access right a to subject X and, because of the presence of the copy flag, can transfer this right, with or without copy flag, to another subject.

Rule R1 expresses this capability. A subject would transfer the access right without the copy flag if there were a concern that the new subject would maliciously transfer the right to another subject that should not have that access right. For example, S1 may place read or read* in any matrix entry in the F1 column.

Rule R2 states that if S0 is designated as the owner of objectX, then S0 can grant an access right to that object for any other subject. Rule 2 states that S0 can add any access right to A[S, X] for any S, if S0 has owner access to X.

Rule R3 permits S0 to delete any access right from any matrix entry in a row for which S0 controls the subject and for any matrix entry in a column for which S0 owns the object.

Rule R4 permits a subject to read that portion of the matrix that it owns or controls.

The remaining rules in Table 15.1 govern the creation and deletion of subjects and objects. 

Rule R5 states that any subject can create a new object, which it owns,and can then grant and delete access to the object. 

Under rule R6, the owner of an object can destroy the object, resulting in the deletion of the corresponding column of the access matrix. 

Rule R7 enables any subject to create a new subject; the creator owns the new subject and the new subject has control access to itself. 

Rule R8 permits the owner of a subject to delete the row and column (if there are subject columns) of the access matrix designated by that subject.

The set of rules in Table 15.1 is an example of the rule set that could be defined for an access control system. A transfer-only right could be defined, which results in the transferred right being added to the target subject and deleted from the transferring subject. The number of owners of an object or a subject could be limited to one by not allowing the copy flag to accompany the owner right. The ability of one subject to create another subject and to have owner access right to that subject can be used to define a hierarchy of subjects. Thus, a subject can create another subject with a subset of its own access rights.

Role-Based Access Control

RBAC models define a role as a job function within an organization. RBAC systems assign access rights to roles instead of individual users. In turn, users are assigned to different roles, either statically or dynamically, according to their responsibilities.

The relationship of users to roles is many to many, as is the relationship of roles to resources, or system objects (Figure 15.7). The set of users changes, in some environments frequently, and the assignment of a user to one or more roles may also be dynamic. The set of roles in the system in most environments is likely to be static,with only occasional additions or deletions. Each role will have specific access rights to one or more resources. The set of resources and the specific access rights associated with a particular role are also likely to change infrequently.

We can use the access matrix representation to depict the key elements of an RBAC system in simple terms, as shown in Figure 15.8. The upper matrix relates individual users to roles. Typically there are many more users than roles. Each matrix entry is either blank or marked, the latter indicating that this user is assigned to this role. Note that a single user may be assigned multiple roles and that multiple users may be assigned to a single role.

The lower matrix has the same structure as the DAC matrix, with roles as subjects. In this matrix the entries are the specific access rights enjoyed by the roles. Note that a role can be treated as an object, allowing the definition of role hierarchies. RBAC lends itself to an effective implementation of the principle of least privilege.That is, each role should contain the minimum set of access rights needed for that role. A user is assigned to a role that enables him or her to perform only what is required for that role. Multiple users assigned to the same role enjoy the same minimal set of access rights.

Give the description of the correct name from the category of the Access Control Policy !

  • Discretionary access control (DAC):
    Controls access based on the identity of the requestor and on access rules (authorizations) stating what requestors are (or are not) allowed to do.
  • Mandatory access control (MAC):
    Controls access based on comparing security labels (which indicate how sensitive or critical system resources are) with security clearances (which indicate system entities are eligible to access certain resources).
  • Role-based access control (RBAC):
    Controls access based on the roles that users have within the system and on rules stating what accesses are allowed to users in given roles.

Unix Access Control (copy)

Traditional UNIX File Access Control

Each UNIX user is assigned a unique user identification number (user ID). A user is also a member of a primary group, and possibly a number of other groups, each identified by a group ID. When a file is created, it is designated as owned by a particular user and marked with that user’s ID. It also belongs to a specific group, which initially is either its creator’s primary group or the group of its parent directory if that directory has SetGID permission set. Associated with each file is a set of 12 protection bits. The owner ID, groupID, and protection bits are part of the file’s inode. Nine of the protection bits specify read, write, and execute permission for the owner of the file, other members of the group to which this file belongs, and all other users. These form a hierarchy of owner, group, and all others, with the highest relevant set of permissions being used.

Figure 15.9a shows an example in which the file owner has read and write access; all other members of the file’s group have read access, and users outside the group have no access rights to the file.

The remaining three bits define special additional behavior for files or directories.Two of these are the “set user ID” (SetUID) and “set group ID” (SetGID) permissions.

If these are set on an executable file, the operating system functions as follows.When a user (with execute privileges for this file) executes the file, the system temporarily allocates the rights of the user’s ID of the file creator or the file’s group,respectively, to those of the user executing the file. These are known as the “effective user ID” and “effective group ID” and are used in addition to the “real user ID”and “real group ID” of the executing user when making access control decisions for this program. This change is only effective while the program is being executed.This feature enables the creation and use of privileged programs that may use files normally inaccessible to other users. It enables users to access certain files in a controlled fashion. Alternatively, when applied to a directory, the SetGID permission indicates that newly created files will inherit the group of this directory. The SetUID permission is ignored.

The final permission bit is the “Sticky” bit. When set on a file, this originally indicated that the system should retain the file contents in memory following execution.This is no longer used. When applied to a directory, though, it specifies that only the owner of any file in the directory can rename, move, or delete that file. This is useful for managing files in shared temporary directories.

One particular user ID is designated as superuser. The superuser is exempt from the usual file access control constraints and has system wide access. Any program that is owned by, and SetUID to, the “superuser” potentially grants unrestricted access to the system to any user executing that program. Hence, great care is needed when writing such programs.

A final point to note is that the traditional UNIX file access control scheme implements a simple protection domain structure. A domain is associated with the user, and switching the domain corresponds to changing the user ID temporarily.

Access Control Lists in UNIX

Many modern UNIX and UNIX-based operating systems support access control lists, including FreeBSD, OpenBSD, Linux, and Solaris. FreeBSD allows the administrator to assign a list of UNIX user IDs and groups to a file by using the set facl command. Any number of users and group scan be associated with a file, each with three protection bits (read, write, execute),offering a flexible mechanism for assigning access rights. 

A file need not have an ACL but may be protected solely by the traditional UNIX file access mechanism. FreeBSD files include an additional protection bit that indicates whether the file has an extended ACL.

FreeBSD and most UNIX implementations that support extended ACLs use the following strategy (e.g., Figure 15.9b):

  1. The owner class and other class entries in the 9-bit permission field have the same meaning as in the minimal ACL case.
  2. The group class entry specifies the permissions for the owner group for this file. These permissions  represent the maximum permissions that can be assigned to named users or named groups, other than the owning user. In this latter role, the group class entry functions as a mask.
  3. Additional named users and named groups may be associated with the file,each with a three-bit permission field. The permissions listed for a named user or named group are compared to the mask field. Any permission for the named user or named group that is not present in the mask field is disallowed.

When a process requests access to a file system object, two steps are performed.

Step 1 selects the ACL entry that most closely matches the requesting process. The ACL entries are looked at in the following order: owner, named users, owning or named groups, and others. Only a single entry determines access. 

Step 2 checks if the matching entry contains sufficient permissions. A process can be a member in more than one group; so more than one group entry can match. If any of these matching group entries contain the requested permissions, one that contains the requested permissions is picked (the result is the same no matter which entry is picked). If none of the matching group entries contains the requested permissions, access will be denied no matter which entry is picked.

What is the right order of the classes for the nine of the protection bits in Traditional UNIX approach?

  • Owner class
  • Group class
  • Other class
  • user
  • group
  • other

Operating Systems Hardening (copy)

Operating Systems Hardening

The first critical step in securing a system is to secure the base operating system upon which all other applications and services rely. A good security foundation needs a properly installed, patched, and configured operating system. Unfortunately, the default configuration for many operating systems often maximizes ease of use and functionality, rather than security. Further, since every organization has its own security needs, the appropriate security profile, and hence configuration, will also differ. Appropriate security configuration guides and checklist sexist for most common operating systems, and these should be consulted, though always informed by the specific needs of each organization and their systems. In some cases, automated tools may be available to further assist in securing the system configuration.

The following basic steps should be used to secure an operating system:

  • Install and patch the operating system.
  • Harden and configure the operating system to adequately address the identified security needs of the system by:
    • Removing unnecessary services, applications, and protocols
    •  Configuring users, groups and permissions
    • Configuring resource controls
  • Install and configure additional security controls, such as antivirus, host-based firewalls, and intrusion detection systems (IDS), if needed.
  • Test the security of the basic operating system to ensure that the steps taken adequately address its security needs.

Operating System Installation: Initial Setup and Patching

Operating System Installation: Initial Setup and Patching

System security begins with the installation of the operating system. As we have already noted, a network-connected, unpatched system is vulnerable to exploit during its installation or continued use. Hence it is important that the system not be exposed while it is in this vulnerable state. Ideally new systems should be constructed on a protected network. This may be a completely isolated network, with the operating system image and all available patches transferred to it using removable media such as DVDs or USB drives.

Alternatively, a network with severely restricted access to the wider internet may be used. Ideally it should have no inbound access, and have outbound access only to the key sites needed for the system installation and patching process. In either case, the full installation and hardening process should occur before the system is deployed to its intended, more accessible, and hence vulnerable,location.

The overall boot process must also be secured. This may require adjusting options on, or specifying a password required for changes to, the BIOS code used when the system initially boots. It may also require limiting which media the system is normally permitted to boot from. This is necessary to prevent an attacker from changing the boot process to install a covert hypervisor or to just boot a system of their choice from external media in order to bypass the normal system access controls on locally stored data.

Care is also required with the selection and installation of any additional device driver code, since this executes with full kernel level privileges, but is often supplied by a third party. The integrity and source of such driver code must be carefully validated given the high level of trust it has.

Note that on change-controlled systems, you should not run automatic updates, because security patches can, on rare but significant occasions, introduce instability.For systems on which availability and uptime are of paramount importance, therefore, you should stage and validate all patches on test systems before deploying them in production.

Remove Unnecessary Services, Application, and Protocols

The default configuration for most distributed systems is set to maximize ease of use and functionality, rather than security. When performing the initial installation,the supplied defaults should not be used, but rather the installation should be customized so that only the required packages are installed. If additional packages are needed later, they can be installed when they are required. 

[NIST08] and many of the security-hardening guides provide lists of services, applications, and protocols that should not be installed if not required.[NIST08] also states a strong preference for not installing unwanted software,rather than installing and then later removing or disabling it. They argue this preference because they note that many uninstall scripts fail to completely remove all components of a package. They also note that disabling a service means that while it is not available as an initial point of attack, should an attacker succeed in gaining some access to a system, then disabled software could be reenabled and used to further compromise a system. It is better for security if unwanted software is not installed, and thus not available for use at all.

Configure Users, Groups, and Authentication

Not all users with access to a system will have the same access to all data and resources on that system. All modern operating systems implement access controls to data and resources. Nearly all provide some form of discretionary access controls.Some systems may provide role-based or mandatory access control mechanisms as well.

The system planning process should consider the categories of users on the system, the privileges they have, the types of information they can access, and how and where they are defined and authenticated. Some users will have elevated privileges to administer the system; others will be normal users, sharing appropriate access to files and other data as required; and there may even be guest accounts with very limited access.

Some operating systems provide special tools or access mechanisms to assist administrative users to elevate their privileges only when necessary, and to appropriately log these actions.One key decision is whether the users, the groups they belong to, and their authentication methods are specified locally on the system, or will use a centralized authentication server. Whichever is chosen, the appropriate details are now configured on the system.

Also at this stage, any default accounts included as part of the system installation should be secured. Those which are not required should be either removed or at least disabled. System accounts that manage services on the system should be set so they cannot be used for interactive logins. And any passwords installed by default should be changed to new values with appropriate security.

Any policy that applies to authentication credentials, and especially to password security, is also configured. This includes details of which authentication methods are accepted for different methods of account access. And it includes details of the required length, complexity, and age allowed for passwords.

Configure Resource Controls

Once the users and their associated groups are defined, appropriate permissions can be set on data and resources to match the specified policy. This may be to limit which users can execute some programs, especially those that modify the system state, or to limit which users can read or write data in certain directory trees. 

Install Additional Security Controls

Further security improvement may be possible by installing and configuring additional security tools such as antivirus software, host-based firewalls, IDS or IPS software, or application white-listing. Antivirus products have traditionally been used on Windows systems,since their high use made them a preferred target for attackers.

Host-based firewalls, IDS, and IPS software also may improve security by limiting remote network access to services on the system. If remote access to a service is not required, though some local access is, then such restrictions help secure such services from remote exploit by an attacker. Firewalls are traditionally configured to limit access by port or protocol, from some or all external systems. Some may also be configured to allow access from or to specific programs on the systems, to further restrict the points of attack, and to prevent an attacker installing and accessing their own malware.IDS and IPS software may include additional mechanisms such as traffic monitoring or file integrity checking to identify and even respond to some types of attack.

Another additional control is to white-list applications. This limits the programs that can execute on the system to just those in an explicit list. Such a tool can prevent an attacker installing and running their own malware, and was the last of the four key DSD mitigation strategies.

Test the System Security

The final step in the process of initially securing the base operating system is security testing. The goal is to ensure that the previous security configuration steps are correctly implemented and to identify any possible vulnerabilities that must be corrected or managed.

There are also programs specifically designed to review a system to ensure that a system meets the basic security requirements, and to scan for known vulnerabilities and poor configuration practices. This should be done following the initial hardening of the system, and then repeated periodically as part of the security maintenance process.

Where is the ideal construction of the operating system?

  • On a protected network
  • On an unprotected network
  • On a network that should have inbound access and have no outbound access

Arrange the users in the system, starting from the user who has the highest privileges and ending with the user who has the lowest privileges.

Chose the right answer! :)

1. (the highest) 

2. (medium)  

3. (the lowest) 

Security Maintenance (copy)


Once the system is appropriately built, secured, and deployed, the process of maintaining security is continuous. This results from the constantly changing environment,the discovery of new vulnerabilities, and hence exposure to new threats. [NIST08] suggests that this process of security maintenance includes the following additional steps:

  •  Monitoring and analyzing logging information
  • Performing regular backups
  • Recovering from security compromises
  • Regularly testing system security
  • Using appropriate software maintenance processes to patch and update all critical software, and to monitor and revise configuration as needed


Logging is a reactive control that can only inform you about bad things that have already happened. But effective logging helps ensure that in the event of a system breach or failure, system administrators can more quickly and accurately identify what happened and thus most effectively focus their remediation and recovery efforts. The key is to ensure you capture the correct data in the logs and then appropriately monitor and analyze this data. 

Logging can generate significant volumes of information. It is important that sufficient space is allocated for them. A suitable automatic log rotation and archive system should also be configured to assist in managing the overall size of the logging information.

Manual analysis of logs is tedious and is not a reliable means of detecting adverse events. Rather, some form of automated analysis is preferred, as it is more likely to identify abnormal activity.

Data Backup and Archive

Performing regular backups of data on a system is another critical control that assists with maintaining the integrity of the system and user data. There are many reasons why data can be lost from a system, including hardware or software failures,or accidental or deliberate corruption. There may also be legal or operational requirements for the retention of data.

Backup is the process of making copies of data at regular intervals, allowing the recovery of lost or corrupted data over relatively short time periods of a few hours to some weeks. 

Archive is the process of retaining copies of data over extended periods of time, being months or years, in order to meet legal and operational requirements to access past data. 

These processes are often linked and managed together, although they do address distinct needs.The needs and policy relating to backup and archive should be determined during the system planning stage. Key decisions include whether the backup copies should be kept online or offline, and whether copies should be stored locally or transported to a remote site. The trade-offs include ease of implementation and cost verses greater security and robustness against different threats.

What is  data backup?

  • Backup is the process of making copies of data at regular intervals, allowing the recovery of lost or corrupted data over relatively short time periods of a few hours to some weeks.
  • Backup is the process of retaining copies of data over extended periods of time, being months or years, in order to meet legal and operational requirements to access past data.

What is data archive?

  • Archive is the process of retaining copies of data over extended periods of time, being months or years, in order to meet legal and operational requirements to access past data.
  • Archive is the process of making copies of data at regular intervals, allowing the recovery of lost or corrupted data over relatively short time periods of a few hours to some weeks.

Windows Security (copy)

Access Control Scheme

Windows provides a uniform access control facility that applies to processes,threads, files, semaphores, windows, and other objects. Access control is governed by two entities: an access token associated with each process and a security descriptor associated with each object for which inter process access is possible. 

Access control scheme

When a user logs on to a Windows system, Windows uses a name/password scheme to authenticate the user. If the logon is accepted, a process is created for the user and an access token is associated with that process object. The access token, include a security ID (SID), which is the identifier by which this user is known to the system for purposes of security. The token also contains SIDs for the security groups to which the user belongs. If the initial user process spawns a new process, the new process object inherits the same access token.

The access token serves two purposes:

  1. It keeps all necessary security information together to speed access validation.When any process associated with a user attempts access, the security subsystem can make use of the token associated with that process to determine the user’s access privileges.
  2. It allows each process to modify its security characteristics in limited ways without affecting other processes running on behalf of the user.

The access token indicates which privileges a user may have. If one of the user’s processes needs to perform a privileged operation,the process may enable the appropriate privilege and attempt access. It would be undesirable to share the same token among all of the user’s processes, because in that case enabling a privilege for one process enables it for all of them.

When an application opens a reference to a securable object, Windows verifies that the object’s security descriptor grants the process the requested access. If the check succeeds, Windows caches the resulting granted access rights.An important aspect of Windows security is the concept of impersonation,which simplifies the use of security in a client/server environment. If client and server talk through an RPC connection, the server can temporarily assume the identity of the client so that it can evaluate a request for access relative to that client’s rights. After the access, the server reverts to its own identity.

Access Token

Figure 15.10a shows the general structure of an access token, which includes the following parameters:

  • Security ID: Identifies a user uniquely across all of the machines on the network.This generally corresponds to a user’s logon name. Special user SIDs were added in Windows 7 for use by processes and services.  These specially managed SIDs are designed for secure management; they do not use the ordinary password polices human accounts do.

  • Group SIDs: A list of the groups to which this user belongs. A group is simply a set of user IDs that are identified as a group for purposes of access control.Each group has a unique group SID. Access to an object can be defined on the basis of group SIDs, individual SIDs, or a combination. There is also an SID which reflects the process integrity level (low, medium, high, or system).
  •  Privileges: A list of security-sensitive system services that this user may call,for example, CreateToken. Another example is the SetBackupPrivilege; users with this privilege are allowed to use a backup tool to back up files that they normally would not be able to read.
  • Default owner: If this process creates another object, this field specifies the owner of the new object. Generally, the owner of a new object is the same as the owner of the spawning process. However, a user may specify that the default owner of any processes spawned by this process is a group SID to whichthis user belongs.
  • Default ACL: This is an initial list of protections applied to the objects that the user creates. The user may subsequently alter the ACL for any object that it owns or that one of its groups owns.

Security Descriptors

Security Descriptor

Figure 15.10b shows the general structure of a security descriptor, which includes the following parameters:

  •  Flags: Define the type and contents of a security descriptor. They indicate whether or not the SACL and DACL are present, whether or not they were placed on the object by a defaulting mechanism, and whether the pointers in the descriptor use absolute or relative addressing. Relative descriptors are  objects that are transmitted over a network, such as information transmitted in an RPC.

  • Owner: The owner of the object can generally perform any action on the security descriptor. The owner can be an individual or a group SID. The owner has the authority to change the contents of the DACL.
  • System access control list (SACL): Specifies what kinds of operations on the object should generate audit messages. An application must have the corresponding privilege in its access token to read or write the SACL of any object.This is to prevent unauthorized applications from reading SACLs (thereby learning what not to do to avoid generating audits) or writing them (to generate many audits to cause an illicit operation to go unnoticed). The SACL also specifies the object integrity level. Processes cannot modify an object unless the process integrity level meets or exceeds the level on the object.
  • Discretionary access control list (DACL): Determines which users and group scan access this object for which operations. It consists of a list of access control entries (ACEs).

When an object is created, the creating process can assign as owner its own SID or any group SID in its access token. The creating process cannot assign an owner that is not in the current access token. The reason for the restriction is to prevent a user from covering his or her tracks after attempting some unauthorized action.

Access control lists

Access control lists are at the heart of the Windows access control facility (Figure 15.10c). Each list consists of an overall header and a variable number of access control entries.Each entry specifies an individual or a group SID and an access mask that defines the rights to be granted to this SID. When a process attempts to access an object,the object manager in the Windows Executive reads the SID and group SIDs from the access token along with the integrity level SID.

If the access requested includes modifying the object, the integrity level is checked against the object integrity level in the SACL. If that test passes, the object manager then scans down the object’s DACL. If a match is found—that is, if an ACE is found with an SID that matches one of the SIDs from the access token—then the process can have the access rights specified by the access mask in that ACE.

Access mask

Figure 15.11 shows the contents of the access mask. The least significant 16 bits specify access rights that apply to a particular type of object.

The most significant 16 bits of the mask contain bits that apply to all types of objects. 

Standard access types

Five of these are referred to as standard access types:

  • Synchronize: Gives permission to synchronize execution with some event associated with this object. In particular, this object can be used in a wait function.
  • Write_owner: Allows a program to modify the owner of the object. This is useful because the owner of an object can always change the protection on the object. (The owner may not be denied Write DAC access.)
  • Write_DAC: Allows the application to modify the DACL and hence the protection on this object
  • Read_control: Allows the application to query the owner and DACL fields of the security descriptor of this object
  • Delete: Allows the application to delete this object

The high-order half of the access mask also contains the four generic access types. These bits provide a convenient way to set specific access types in a number of different object types.

Generic access types

The high-order half of the access mask also contains the four generic access types. These bits provide a convenient way to set specific access types in a number of different object types. That is the purpose of the generic access bits,which are as follows:

  • Generic_all: Allows all access
  • Generic_execute: Allows execution if executable
  • Generic_write: Allows write access
  • Generic_read: Allows read-only access

The generic bits also affect the standard access types.

Access_system_security bit

The remaining two bits in the access mask have special meanings. The Access_System_Security bit allows modifying audit and alarm control for this object. However, not only must this bit be set in the ACE for an SID but the access token for the process with that SID must have the corresponding privilege enabled.

Maximum_Allowed bit

Finally, the Maximum_Allowed bit is not really an access bit, but a bit that modifies the algorithm for scanning the DACL for this SID. Normally, Windows will scan through the DACL until it reaches an ACE that specifically grants (bit set) or denies (bit not set) the access requested by the requesting process or until it reaches the end of the DACL. The Maximum_Allowed bit allows the object’s owner to define a set of access rights that is the maximum that will be allowed to a given user.

With this in mind, suppose that an application doesnot know all of the operations that it is going to be asked to perform on an objectduring a session. There are three options for requesting access:

  1. Attempt to open the object for all possible accesses. The disadvantage of this approach is that access may be denied even though the application may have all of the access rights actually required for this session.
  2. Only open the object when a specific access is requested, and open a new handle to the object for each different type of request. This is generally the preferred method because it will not unnecessarily deny access, nor will it allow more access than necessary. In many cases the object itself does not need to be referenced a second time, but the DuplicateHandle function can be used to make a copy of the handle with a lower level of access.
  3. Attempt to open the object for as much access as the object will allow this SID. The advantage is that the client application will not be artificially denied access, but the application may have more access than it needs. This latter situation may mask bugs in the application.

The Access Token serves two purposes:

  • It keeps all necessary security information together to speed access validation. When any process associated with a user attempts access, the security subsystem can make use of the token associated with that process to determine the user’s access privileges.
  • It allows each process to modify its security characteristics in limited ways without affecting other processes running on behalf of the user.
  • The access token indicates which privileges a user may not have.
  • The Access token gives new privileges to the user who previously did not have access to any of the processes.

What are the parameters that belong to the Access Token?

Hint: Chose 5 of the following parameters...

  • Flags
  • Owner
  • Security ID (SID)
  • Group SIDs
  • Privileges
  • Default owner
  • System access control list
  • Discretionary access control list
  • Default ACL

What are the parameters that belong to the Security Descriptor?

Hint: Chose 4 of the following parameters...

  • Security ID (SID)
  • Group SID
  • Flags
  • Ownes
  • Privileges
  • Default owner
  • Default ACL
  • System access control list
  • Discretionary access control list

What does the bits in the access mask mean?

  • Specific access types
  • Standard access types
  • Generic access types
  • Access system security
  • Maximum allowed

Summary (copy)


The scope of operating system security is broad. This chapter focuses on some of the most important topics. The most prominent issue for OS security is countering thread from intruders and malicious software. Intruders attempt to gain unauthorized access to system resources, while malicious software is designed to penetrate system defenses and become executable on target systems. Countermeasures to both types of threat include intrusion detection systems, authentication protocols, access control mechanisms, and firewalls. 

One of the most common techniques for compromising OS security is the buffer overflow attack. A condition at an interface under which more input can be placed into a buffer or data-holding area than the capacity allocated, overwriting other information. Attackers exploit such a condition to crash a system or to insert specially crafted code that allows them to gain control of the system. System designers use a variety of compile-time and runtime defenses to counter this type of attack.

Another important area of security defense is access control. Access control measures include those that secure access to file system and to the OS user interface.Traditional techniques for access control are referred to as discretionary access control.A more flexible approach that has gained considerable support is role-based access control, in which access depends not only on the identity of the user but on the specific role that user can assume for a specific task or set of tasks.