Security problem

Vitalize cross-team communication, collaboration and enhance threat modeling process

One of the challenges of cross-team communication in agile world is for non-technical teams such as product management, operations, or even customers to understand the context of security testing. Besides the comprehensive understanding of how security has been tested it is also essential to verify that all potential threats are identified and are properly addressed.

The requirement is to make security testing more transparent for all project stakeholders and at the same time through better collaboration and communication to write threat stories to express functional and non-functional security requirements. By encouraging conversations across various stakeholders, it is assured that the most important concepts and aspects of the developing application or system are discussed through concrete, real-life examples of expected requirements and product behavior.

Additionally, in DevOps and especially in DevSecOps world it is critical for security testing not to produce bottlenecks or delays. Delays in security testing greatly increase the risk of identifying a security vulnerability or flaw that will delay a release schedule. Security testing practices should merge well with automation processes and security testing tools should be used on top of automation testing frameworks in efficient ways. The requirement is to find how free and open source tools can be bundled together with various common security tools to create a framework that incorporates automated security into DevOps pipeline.

Security control proposal

From TDD to ATDD and BDD

Test-Driven Development (TDD) is a testing methodology implemented from a developer’s perspective and focuses on how functions of the software should work. Since it verifies whether the implementation of a feature/functionality is correct it can be considered unit level testing. In a TDD approach, automated test cases for every small functionality of an application (unit tests) are written before the development of functional code. The code is modified (re-factoring) only when the test case fails.

The TDD methodology involves the following steps:

1. Design and write test cases based on the requirements
2. Execute tests: tests fail
3. Modify/re-factor the code for the test to pass successfully
4. Repeat tests: tests pass

A step beyond TDD

Acceptance Tests are the final level of testing which verifies whether the system’s functionality matches the specification.

In Acceptance Test-Driven Development (ATDD) technique, a single acceptance test is written from the user’s perspective. It mainly focuses on satisfying the functional behavior of the system. ATDD is a process in which high-level acceptance tests are automated to fail and then production code is developed make the acceptance test pass. ATDD focusses on the code meeting the requirements of the software

Key practices in ATDD:

• Analyze real-world scenarios.
• Define the acceptance criteria for the scenarios.

The most popular acceptance testing automation framework is Robot Framework - a keyword driven testing acceptance automation framework. Robot Framework is programming language independent though the Robot Framework itself was built by Python.

Business-Driven Development (BDD) is a testing approach derived from the Test-Driven Development (TDD) methodology. BDD is an abstraction of TDD. In BDD, tests focus on the behavior of the application and the requirements can be defined in a standard approach using simple English language. The goal of BDD methodology is to promote a shared understanding between tech and non-tech teams as early as possible in user stories lifecycle. BDD is effectively a communication tool, allowing the business and security experts to define functional and non-functional behavior in a natural language, while still allowing that behavior to be captured using automated tests.

Key practices in BDD:

• Describe Feature in high level (note: Features are usually technology agnostic).
• Create user stories for each feature to express requirements.
• Decompose user stories into Scenarios.
• Create Steps to test Scenarios.

Even if in some cases BDD and ATTD terms are used interchangeably ATDD is a technique used WITHIN BDD. They are not one and the same. BDD leverages ATDD technique along with other techniques around conversations, collaboration and automation to ensure that a team delivers business goals. A key difference between them is that BDD focuses more on the behavior of the feature, whereas ATDD focuses on capturing the accurate requirements. BDD focuses on solving potential issues of delivering the right content according to the requirements rather than focuses on the automation techniques.

The following table lists the key differences between TDD, ATDD and BDD.

Method	TDD	BDD	ATDD
Definition	TDD is a development technique that focuses more on the implementation of a feature	BDD is a development technique that focuses on the system’s behavior	ATDD is a technique similar to BDD focusing more on capturing the requirements
Main Focus	Unit Tests	Understanding Requirements	Writing Acceptance Tests

The Gherkin language

BDD approach defines the Given-When-Then approach to write test cases and develop a feature based on its behavior. In this way BDD provides a dynamic documentation for the whole project cycle.

• The Gherkin language is the Core of any BDD Framework.
• It is a Domain Specific, Business Readable language.
• Enables all stakeholders to understand the software without a deep technical understanding of the implementation.
• In the BDD world, the Test Cases are broken down into the Gherkin Scenarios.
• The Scenarios are then implemented using the Gherkin’s G-W-T construct: - Given – there is some context - When – some action occurs - Then – some result is expected language

Notepad++ through User Defined Language option, allows users to customize the syntax highlighting for any language.

Below is an example from Gherkin syntax highlighting in Notepad++:

Source: Automation Panda Github repository– the file is named gherkin_npp_udl.xml.

BDD Testing Frameworks

The following table lists the key usages of some common BDD testing frameworks:

BDD framework	Usage and scenarios
Robot Framework(*)	It’s a common keyword driven testing acceptance automation framework.The Robot Framework is programming language independent though the Robot Framework itself was built by Python
Behave	It’s a Python BDD framework
JGiven	It’s a Java BDD framework
Gauntlt	It’s a purpose-built for security BDD framework in Ruby

(*) As it was explained in ATTD section because of its keyword driven nature Robot Framework can be also considred an ATDD framework.

Other common automation frameworks that apply to security testing are:

• Selenium Web UI framework is used to walk through the UI flow for security tools to inspect security issues.
  For more information about Selenium browser automation, see: Selenium,Selenium documentation
• JMeter can be used with security payloads to do RESTful API security testing.
• Robot Framework can integrate with OWASP ZAP to introduce BDD testing into the security testing cycle.

Applied scenarios:

API testing	Web UI automation
Web/RESTful API security testing • Parameterized (data-driven) with Fuzz testing	• Login with different users or wrong accounts • Logout users for session management testing • Create a new user account • Brute force user account login
• Robot Framework requests library • JMeter • FuzzDB • OWASP ZAP	• Robot Framework • Selenium • OWASP ZAP

BDD Security Testing Frameworks

BDD security testing frameworks aim to integrate security testing tools and to define the security testing cases by using human-readable user-story statements. BDD security testing frameworks consolidate the security tools (e.g. Nmap, SSLyze, SQLmap, ZAP etc.) and require each security testing tool to perform the execution based on the defined user stories.

The following table lists the key differences between three open source BDD security testing frameworks. These frameworks are flexible enough to perform security tool integration and to provide consolidated testing results.

BDD Security Frameworks	MITTN	GAUNTLT	BDD Security
Programming language	Python	Ruby	Java
BDD framework	Behave	Cucumber	Cucumber Selenium
Windows/Unix	Unix	Both	Both
Default plugins (security tools)	Burpsuite SSLyze Radamsa API fuzzing	CURL NMAP SSLyze SQLmap Garm Heartbleed dirb Arachi Generic command line	OWASP ZAP SSLyze Nessus Nmap

Reference implementation

Practicing BDD in security

Goals:

• Make security testing more transparent for all project stakeholders.
• Define a set of concrete use case that illustrate key outcomes of security features.
• Fit security requirements into User stories: pay extra attention to the acceptance criteria.
• Write test cases that focus on application behavior: listen on the right ports; correct authentication patterns; correct use of cryptography etc.
• Use security testing tools on top of the BDD automation testing frameworks to test security features.

Write Security Specifications

• All stakeholders (product/business owners, developers, testers, security experts) work together to define stories and scenarios of what a specific security feature shall deliver.
• Stories and scenarios form the basis of the specifications and act also as acceptance criteria. Turn acceptance criteria into automated acceptance tests = executable specifications.

Write threat stories

• Threat modelling is all about having a conversation, shared understanding, action. This BDD!!
• Describe the attack from an adversary perspective.
• Explain why an adversary would carry out the attack.
• Provide scenario(s) that describe way(s) the attack can occur.

BDD in action

• The feature files are used to test various parts of the project/solution.
• The steps files contain the BDD steps for the feature files.

Examples:

Example 1 – Threat Story: SSH brute force attack

Feature: SSH brute force attack
    As an attacker in order to get shell access to a system I want that system    to use weak authentication

Scenario: Online password brute force
    Given a system running an accessible SSH daemon
    And a wordlist of users
    And a wordlist of passwords
    When we SSH to the system using each combination of username and password
    Then we should get shell access to the system

Example 2 – Security Requirement: only required ports are open

Feature: Ensure that only the ports for the application to function are open
    As system administrator I want to ensure that only ports described in application’s communication matrix are open.

Scenario: Administration ports
    Given NMAP is installed
    When I launch an NMAP network scan "nmap -F host"
    Then the output should match "22 tcp open"
    And the output should not match "23 tcp open"

Example 3 - OWASP Application Security Verification Standard 4.0

Feature: REST services accept only valid requests

As system administrator I want to verify that REST services explicitly check the incoming Content-Type to be the expected one, such as application/xml or application/JSON (OWASP/ASVS 13.2.6)

Scenario: POST xml
    Given I Set POST api endpoint
    When I Set HEADER param request content type as "application/xml." 
    And Set request Body
    And Send a POST HTTP request 
    Then I receive valid HTTP response code 201
    And Response BODY "POST" is non-empty

Scenario: POST JSON
    Given I Set POST api endpoint
    When I Set HEADER param request content type as "application/json." 
    And Set request Body
    And Send a POST HTTP request 
    Then I receive valid HTTP response code 201
    And Response BODY "POST" is non-empty. 

Reject requests containing unexpected or missing content type headers with HTTP response status 406 Unacceptable or 415 Unsupported Media Type

Scenario: POST Invalid
    Given I Set POST  api endpoint
    When I Set HEADER param request content type as NOT application/json OR application/xml
    When I Set HEADER param request content type as " text/javascript."
    When I Set HEADER param request content type as " application/gzip."
    And Set request Body
    And Send a POST HTTP request
    Then I receive valid HTTP response code 406 or 415
    And Response BODY "POST" is non-empty

Incorporate BDD into Software/System/Service Development

Process Example:

Practical implementation

Gauntlt and Robot Framework

The following practical examples demonstrate automated security testing against the Damn Vulnerable Web App(DVWA) web application (http://www.dvwa.co.uk/) using Gauntlt and Robot Framework. DVWA is installed locally on a Linux based platform.

The security testing scenarios cover:

• Use NMAP with Gauntlt and Robot Framework.
• Use sqlmap with Robot Framework.

Gauntlt is a free and open source tool which bundles together various common security tools and can add more to further customize a test suite. Robot Framework is open and extensible and can be integrated with virtually any other tool to create powerful and flexible automation solutions. Robot Framework uses a keyword-driven approach to define the testing steps and Gauntlt uses a GWT approach to define the testing scripts. The following table lists the main differences between those two testing frameworks.

Framework	Robot Framework	Gauntlt
Type	ATDD/BDD	BDD
Script format	Keyword-driven script: • Execute command • Create session • Should not contain	Scenario-driven script: •Given.. • When… • Then the output should…
Integration with security tools	No additional security adapters are needed. Robot Framework uses one of the following approaches to communicate with security tools: • command-line interface (CLI): Execute command • REST API: Requests library	Gauntlt requires security adapters, although it provides a generic command-line adapter that can be used for most CLI security tools Gauntlt includes attack adapters for these tools: • curl • nmap • sslyze

NMAP BDD testing with Gauntlt

Port Scanning

The Nmap Guantlt script is defined as follows, with the filename nmap_port_scan_gauntlt.feature:

Execute the script:

gauntlt nmap_port_scan_gauntlt.feature

The following screenshot shows the testing results for Gauntlt.

Nmap NSE testing

The NMAP Guantlt script for SQL Injection attacks and weak TLS is defined as follows, with the filename nmap-nse-web-attacks.feature:

Execute the script: gauntlt nmap-nse-web-attacks.feature

The following screenshot shows one of the testing results for Gauntlt.

NMAP and sqlmap BDD with Robot Framework

Test setup:

Robot Framework key component	Use in security testing scenario
Run Process	Use Run Process to execute the security testing tools
${result.stdout}	Run Process will store all the output into this variable, which can be used to verify the test results
Log	Use ‘Log’ to print the command execution results in the report. This will be the NMAP and sqlmap console output results

Nmap with Robot Framework

Scenario: Verify that telnet port is not open

Robot Framework steps:

Execute and review the results:

robot    nmap_scan.robot

The following screenshot shows the test results for robot framework.

Here is one of the Robot framework HTML reports, log.html

Sqlmap with Robot Framework

Scenario: Verify Database Management Accounts

Robot Framework steps:

Execute and review the results:

robot SQL_test.robot

The following screenshot shows the test results for robot framework.

Here the Robot framework HTML reports, log.html. In the second screenshot DB accounts are displayed.

Sources:

1. OWASP cloud security project
   
2. OWASP Cloud Security Workshop material
   
3. Behavior Driven Development (BDD) and Continuous Integration & Delivery (CI-CD)- Rebaca Technologies
   
4. Practical Security Automation and Testing by Tony Hsiang-Chih Hsu