Security Testing Tutorial

Rahul — Sat, 28 Oct 2023 04:05:33 +0000

What is Security Testing?

Security testing is a process of evaluating a software application, system, or network to identify and rectify vulnerabilities and weaknesses that could lead to unauthorized access, data breaches, or system compromises.

Why is Security Testing Important?

Security testing is crucial because it helps:

Protect sensitive data
Ensure the confidentiality, integrity, and availability of the system
Comply with industry standards and regulations
Prevent financial loss and damage to reputation

Types of Security Testing

Penetration Testing

Penetration testing, often referred to as “ethical hacking,” involves simulating real attacks to identify vulnerabilities. Skilled testers attempt to exploit weaknesses to demonstrate potential risks.

Vulnerability Scanning

Vulnerability scanning uses automated tools to identify known vulnerabilities in your system or network. It’s an efficient way to detect low-hanging fruit.

Security Auditing

Security auditing involves reviewing system configurations, access controls, and policies to identify security weaknesses. This can be done manually or with the help of tools.

Security Code Review

Security code review involves examining the source code for security flaws, such as injection vulnerabilities, insecure data storage, and authentication issues. Tools like static code analyzers can help.

Threat Modeling

Threat modeling is a proactive approach to security. It identifies potential threats and vulnerabilities during the design phase, allowing you to mitigate them early.

Security Testing Process

Planning

Define the scope, objectives, and methodologies of your security testing. Establish timelines and gather the necessary resources.

Scanning

Perform security scans and tests using the selected tools and methodologies. This may include penetration testing, vulnerability scanning, and code reviews.

Analysis

Examine the results of your tests, prioritize vulnerabilities, and assess their potential impact. Determine whether they are false positives or actual threats.

Reporting

Create a comprehensive report detailing the vulnerabilities, their severity, and recommendations for remediation. Share this with the development and operations teams.

Remediation

Collaborate with developers and administrators to fix identified vulnerabilities. Ensure proper testing and validation of fixes before deploying updates.

Security Testing Best Practices

Define a Security Testing Strategy

Establish clear objectives, scope, and methodologies for your security testing. Consider the critical areas to focus on based on your application’s functionality.

Understand Your Application

Comprehend the architecture, components, and data flows in your application. Knowing how your system works is essential for identifying potential vulnerabilities.

Create Test Cases

Develop comprehensive test cases that cover different aspects of security, such as authentication, authorization, data validation, and encryption.

Use Security Testing Tools

Leverage security testing tools to automate scans and assessments. Some popular tools include OWASP ZAP, Nessus, Burp Suite, Metasploit, and Wireshark.

Test for OWASP Top 10 Vulnerabilities

Pay special attention to the OWASP Top 10, a list of the most critical web application security risks. These include issues like injection attacks, broken authentication, and security misconfigurations.

Conduct Regular Testing

Security is an ongoing process. Schedule regular security tests to ensure that new vulnerabilities are not introduced as your application evolves.

How to perform security testing parallel in each stage of the software development life cycle(SDLC)

Performing security testing in parallel with each stage of the Software Development Life Cycle (SDLC) is a best practice known as “Shift Left” security. This approach ensures that security is considered and integrated from the early stages of development rather than being tacked on at the end. Here’s how you can perform security testing at each stage of the SDLC:

Requirements Phase:
Threat Modeling: Identify potential security threats and vulnerabilities based on project requirements. Use threat modeling tools to visualize and analyze the threat landscape.
Design Phase:
Security Architecture Review: Assess the security of the system’s architecture and design. Ensure that proper security measures, such as authentication, authorization, and encryption, are incorporated into the design.
Development Phase:
– Static Application Security Testing (SAST): Use SAST tools to analyze the source code for security vulnerabilities. Developers can receive early feedback on issues they introduce.
– Code Review: Conduct manual code reviews to identify security flaws, such as injection vulnerabilities, authentication issues, and insecure data handling.
Testing Phase:
– Dynamic Application Security Testing (DAST): Perform dynamic scanning of the running application to find vulnerabilities like input validation issues, session management problems, and other runtime issues.
– Penetration Testing: Simulate real-world attacks to identify vulnerabilities that automated testing might miss. This stage helps validate the effectiveness of security controls.
Deployment Phase:
Security Configuration Audit: Ensure that the deployed system is configured securely. This includes reviewing server configurations, database settings, and access controls.
Operations and Maintenance Phase:
– Continuous Monitoring: Implement tools for continuous security monitoring to detect and respond to new vulnerabilities and threats as they emerge.
– Incident Response: Develop and regularly update an incident response plan to react to security incidents promptly.
Post-Release Phase:
Security Patch Management: Continuously monitor for security updates and apply patches to address vulnerabilities as they are discovered.
User Training and Awareness
Educate end-users and administrators about security best practices and the importance of maintaining a secure environment.

To perform security testing effectively in parallel with each stage of the SDLC:
Integrate Security into Workflows: Incorporate security checks and approvals into your development and release pipelines. Automated tools, like CI/CD security plugins, can help with this.
Collaboration: Promote collaboration between security teams and development teams. Security experts should work closely with developers to provide guidance and support.
Security Standards and Guidelines: Establish security coding standards and guidelines that developers should follow throughout the development process.
Automate Security Testing: Utilize automated security testing tools to ensure consistency and efficiency in identifying vulnerabilities.
Documentation and Reporting: Maintain detailed documentation of security findings and regularly communicate security status to relevant stakeholders.
Regular Training: Provide ongoing training and awareness programs for development and operations teams to keep them updated on the latest security threats and best practices.

Example Test Scenarios for Security Testing

Here are some example test scenarios for security testing, organized by various aspects of security to cover a wide range of potential vulnerabilities and threats:

1. Authentication and Authorization:

1.1. Authentication Bypass:

Verify that an unauthenticated user cannot access restricted areas or features.
Attempt to log in with incorrect credentials and check if the system prevents unauthorized access.

1.2. Password Policy:

Test the system to ensure it enforces password complexity rules.
Verify that users are prompted to change their passwords periodically.

1.3. Session Management:

Log in with one account, and then attempt to access another user’s account to ensure proper session isolation.
Check for session timeouts and session fixation vulnerabilities.

2. Input Validation:

2.1. SQL Injection:

Attempt to inject SQL code into input fields to verify that the system prevents SQL injection attacks.

2.2. Cross-Site Scripting (XSS):

Try injecting malicious scripts into user inputs to check if the application escapes or filters out dangerous script code.

2.3. Cross-Site Request Forgery (CSRF):

Craft a request that tricks an authenticated user into executing unwanted actions, such as changing their password or sending funds.

3. Data Security:

3.1. Data Encryption:

Verify that sensitive data, such as passwords and payment information, are properly encrypted during transmission (using HTTPS) and storage.

3.2. Data Leakage:

Attempt to access restricted data by manipulating URLs, cookies, or hidden fields.
Verify that confidential data is not exposed in error messages.

4. Configuration Management:

4.1. Security Misconfiguration:

Check for default configurations and ensure they have been changed to more secure settings.
Verify that unnecessary services and ports are not exposed.

4.2. File Uploads:

Attempt to upload malicious files and verify that the system correctly identifies and blocks them.
Ensure that uploaded files are stored securely and cannot be executed on the server.

5. Access Control:

5.1. Unauthorized Access:

Check if regular users can access administrative functions.
Verify that users can only view and modify data they are authorized to access.

5.2. Role-Based Access Control:

Confirm that different roles have the appropriate permissions, and users cannot escalate their privileges.

6. Error Handling:

6.1. Detailed Error Messages:

Attempt to cause errors in the application and verify that detailed error messages do not reveal sensitive information about the system.

7. Business Logic:

7.1. Business Logic Flaws:

Test the application for logical vulnerabilities, such as order manipulation, price changes, or access to unauthorized resources.

8. API Security:

8.1. API Authentication and Authorization:

Ensure that APIs are properly protected with authentication and authorization mechanisms.
Test for insecure direct object references.

8.2. API Rate Limiting and Throttling:

Verify that APIs have rate limiting to prevent abuse.

Security Testing Tools

Web Application Security Testing:

OWASP ZAP (Zed Attack Proxy): An open-source web application security scanner and proxy tool. It helps you find security vulnerabilities in web applications during development and testing.
Burp Suite: A comprehensive tool for web application security testing, including scanning, crawling, and proxy features. It is widely used by penetration testers and security professionals.
Nessus: A powerful vulnerability scanner that can scan networks for security vulnerabilities and misconfigurations. It provides detailed reports and recommendations for remediation.
Acunetix: A web vulnerability scanner that can detect a wide range of security issues in web applications, including SQL injection, cross-site scripting, and more.
Nikto: An open-source web server scanner that identifies various vulnerabilities and misconfigurations, such as outdated software and security issues.
AppScan (IBM Security AppScan): A tool for web application security testing, offering features like dynamic scanning, static scanning, and interactive scanning.

Network Security Testing:

Wireshark: A widely-used network protocol analyzer that helps you capture and inspect data on a network. It is invaluable for understanding network traffic and diagnosing network issues.
Nmap (Network Mapper): A versatile network scanning tool that can discover open ports, services, and vulnerabilities on remote hosts.
Metasploit: A penetration testing framework that includes a collection of tools for finding and exploiting vulnerabilities. It is often used by security professionals for ethical hacking.

Code Analysis and Static Analysis:

Checkmarx: A static code analysis tool that identifies security vulnerabilities in source code. It’s used to find and remediate security issues early in the development process.
Fortify (Micro Focus Fortify): A suite of tools for static and dynamic application security testing. It helps identify and fix vulnerabilities in code.

Mobile Application Security Testing:

MobSF (Mobile Security Framework): An open-source mobile application security assessment tool that supports Android and iOS applications.
Drozer: A comprehensive Android security testing framework that helps find and exploit vulnerabilities in Android apps.

Cloud Security Testing:

Prowler: A security tool for AWS that performs security best practices checks on your AWS infrastructure.
Terraform Compliance: A tool to check security compliance with Terraform templates, which are used for infrastructure as code in cloud environments.

Database Security Testing:

Sqlmap: An open-source penetration testing tool that automates the process of detecting and exploiting SQL injection vulnerabilities.
DbProtect (Trustwave DbProtect): A database security tool that identifies vulnerabilities and misconfigurations in databases and provides recommendations for securing them.

Operating System and Server Security Testing:

OpenVAS (Open Vulnerability Assessment System): An open-source vulnerability scanner that can scan for vulnerabilities on servers and systems.
Lynis: A security auditing tool for Unix and Linux-based systems. It performs system hardening and compliance testing.

Security testing is an integral part of ensuring the safety and integrity of your software and systems. By following best practices, using appropriate tools, and conducting various types of security testing, you can significantly reduce the risk of security breaches and protect your organization’s valuable assets. Remember that security is an ongoing process, so regularly test and update your security measures to stay ahead of potential threats.

Data Migration Testing | Test Strategy | Testing Phases | Challenges

Rahul — Sun, 30 Aug 2020 03:58:40 +0000

In the last article on Data Migration, we learnt What is the Data Migration, Types of it and various strategies. Now it’s the time to learn about Data Migration Testing.

What is Migration Testing?

Migration Testing is a verification process of migration of the legacy system to the new system with minimal disruption/downtime, with data integrity and no loss of data, while ensuring that all the specified functional and non-functional aspects of the application are met post-migration.
Migration Testing verifies the process of migrating the data from the legacy, or current system, to the new target system. It does so with limited downtime and no loss of data or issues with data integrity. In addition, it does so while making sure functional and non-functional requirements / expectations of the application, are met post-migration.

Why Migration Test?

As we know, the application migration to a new system could be for various reasons, system consolidation, obsolete technology, optimisation, or any other reasons.

Hence while the System in Use needs to be migrated to a new system, it is essential to ensure the below points:

Any kind of disruption/inconvenience caused to the user due to migration needs to be avoided or minimised like downtime, loss of data.
Need to ensure if the user can continue to use all the features of the application.
To ensure compatibility of the new/upgraded application with the hardware and software that the legacy application supports.
To ensure the existing functionalities works as in the legacy application seamlessly.
Critical Defects related to the data and data type need to be identified & fixed during testing.
To ensure no performance degradation post migration.
To ensure if the connection between servers, hardware, software and therefore the Data flow between different components remains intact.

Hence in order to ensure a smooth migration by eliminating the above defects, it is essential to carry out in-depth Migration Testing.

Importance of Data Migration Testing

There are some challenges that could arise during data migration testing, which is precisely why the testing itself is so crucial. Some of these challenges include:

Data corruption
Testing the mapping or path of data.
Unintended data modifications in the legacy system and in the data structures mapping.

By uncovering and resolving these issues during the testing phase can be priceless and a success.

Test Phases in Migration Testing

The different phases of Migration test need to be considered:

1. Pre-Migration Testing
2. Data Cleansing
3. Post Migration Testing

Phase 1 : Pre-Migration testing

Before any migration has taken place, the following testing actions should take place:

Ensure that the scope of the data is clearly understood what data has to be included, what data has to be excluded, which data needs.
Ensure that the destination as well as the load process is understood.
Perform data mapping between legacy and the new application.
Ensure that the data scheme is known including mandatory fields, field names, field types, data types etc. for both the original data source and the destination system.
Understand the data cleansing requirements.
Understand any interfacing with other systems or secondary suppliers of data into the system.
Tables in the legacy system are to be noted down and if any tables are dropped and added post migration needs to be verified.
Records in each table, views should be noted in the legacy application.
Prepare test cases, test scenarios, and use cases for new conditions in the new applications.
Execute a set of test cases, scenarios with a set of users and keep the results, logs stored. The same needs to be verified after Migration to ensure that legacy data and functionality are intact.
Count of the data and records should be noted down clearly, it needs to be verified after Migration for no loss of data.
Ensure the mapping to user interface is correct.
Ensure the mapping to business process is correct.
Test tool configuration and script generation using field details.
Identify data subset (sampling) details for testing.
Ensure all business cases, user stories, or use cases are understood.

Phase 2 : Data Cleansing

It is vitally important that data cleansing is performed on a system that houses and maintains data. The following data cleansing tasks should be planned:

Understand the error types: these may include blank fields, too-long data lengths, or bad characters.
Identify method of checking the data: either interrogation through database using SQL commands or the use of data migration tools.
Understand the data dependencies i.e. if changing the data in one table impacts the data in another linked table.
Fix the data issues.
Verify the count of the data.

Phase 3 : Post-Migration Testing

Following the migration of the application is migrated successfully, Post-Migration testing comes into the picture.

In addition to testing the known business flows, the testers should carry out the following testing, including negative testing approaches, which primarily ensure that data cleansing is being carried out at run time within the system:

Check whether all the data in the legacy is migrated to the new application within the downtime that was planned. Example compare the number of records between legacy and the new application.
Check whether all the schema changes as per the new system are updated.
Data migrated from the legacy to new application should retain its value and format.
Test the migrated data type including business data types and technical data types.
Check the data flow within the application
Check and verify the earlier supported functionality
Check for legacy data’s redundancy. No legacy data should be duplicated itself during migration
Input bad data: attempt to violate the validation rules including the use of boundary value analysis, equivalence partitioning and error guessing.
Bypass mandatory data – attempt to proceed further without filin the mandatory data fields.
Checking data locks: Where data is being written, it should not be possible for multiple users to access the same new record within the database.
Check for database security and data integrity.
Create new users on the system and carry out tests to ensure that functionality is accessible to the newly created user.
Data segregation – Deleting the data in the new application, should not delete data in legacy as well. Similarly, any data addition in the new application should not reflect back on the legacy system.
Performance Testing: To ensure that migration has not degraded the performance of the system and Data is accessible in accordance to the required performance.
Security Testing: Access to the data should be restricted appropriately.
Usability: Verify the ease of Use of the functionalities migrated for the end user
Backward Compatibility Testing: Migration of the system also calls for the testers to verify the ‘Backward Compatibility’, wherein the new system introduced is compatible with the old system.
Rollback Testing – Migration failure test scenarios need to be designed as part of negative testing and rollback mechanism needs to be tested in case of any issues while carrying out the migration or if there is a migration failure at any point of time during migration.

Data Migration Testing Strategy

Designing the test strategy for migration include a set of activities to be performed and few aspects to be considered. This is to minimise the errors and risks that occur as a result of migration and to perform the migration testing effectively.

Activities in this Testing:

Specialized team formation: Form the testing team with the members having the required knowledge & experience and provide training related to the system that is being migrated.
Business risk analysis: Current business should not be hampered after migration and hence carry out ‘Business Risk Analysis’ meetings involving the right stakeholders and identify the risks and the implementable mitigation. The testing should include scenarios to uncover those risks and verify if proper mitigation have been implemented.

Possible errors analysis: Conduct ‘Possible Error Analysis’ using appropriate ‘Error Guessing Approaches’ and then design tests around these errors to unearth them during testing.
Migration scope analysis and identification: Analyse the clear scope of the migration test as when and what needs to be tested.

Identify the appropriate Tool for Migration: While defining the strategy of this testing, automated or manual, identify the tools that are going to be used. e.g. Automated tool to compare source and destination data.
Identify the appropriate Test Environment for Migration:

Identify separate environments for Pre and Post Migration environments to carry out any verification that is required as part of testing. Understand and document the technical aspects of the Legacy and New system of Migration, to ensure that the test environment is set up as per that.

Migration Test Specification Document and review: Prepare Migration Test Specification document which clearly describes the test approach, areas of testing, testing methods i.e. automated, manual, testing methodology Number of cycles of testing, schedule of testing, approach of creating data and using live data , test environment specification, testers qualification etc. and run a review session with the stakeholders.

Production launch of the migrated system: Analyse and document the to-do list for production migration and publish it well in advance.

Challenges in Data Migration Testing

Challenges faced in this testing are mainly with data. Below are few in the list:

Data Quality: We may find that the data used in the legacy application is of poor quality in the upgraded application. In such cases, data quality has to be improved to meet business standards. Some of the ways to resolve this may be execute some mock migrations and pilot migrations before the final migration. It may give some errors which we never expected during any of the previous phases.
Truncation and precision of data: In this case, the data to be migrated is partially moved to the target system. It can be challenging because of the additional padded spaces or a smaller number of bytes in the source data.
Data type and Data Mismatch: Data and Data type migrated from the legacy to the upgraded application may mismatch in the new one. This may be due to the change in data type, format of data storage, the purpose for which the data is being used may be redefined. SO The data types must be mapped correctly.

Data Loss: Data might be lost while migrating from the legacy to the upgraded application. This may be with mandatory fields or non-mandatory fields. If the data lost is for non-mandatory fields, that is less risky as it can be updated again but if the mandatory field’s data is lost, then the record itself becomes void and it cannot be retracted. This might result in data loss and should have to be retrieved.

Null data translation: Null data should be translated to the Target as null itself and not as spaces or default values. The application can have some checks for null validation.
Data Volume: Huge Data that requires a lot of time to migrate within the downtime window of the migration activity.

Extra records:Duplicate records can come from different data sources. So, before loading the data to the target system, a filtering and transformation of the data needs to be done
Simulation of a real-time environment:Simulation of a real-time environment in the testing lab is another real challenge, as testers get into different kind of issues when they test the application / system with the the real data and the real system. So, data sampling, replication of real environment, identification of volume of data involved in migration is quite amportant while carrying out data Migration Testing.

Unclear / Missing requirements: There are instances where some requirements are missed due to various reasons including lack of communication with end users or subject matter experts. If it’s not addressed properly, it affects the data migration project hugely.
Extraneous / Duplicate records:Duplicate records can come from different data sources. So, before loading the data to the target system, a filtering and transformation of the data needs to be done in order to remove the duplicate records.

Final Thoughts

According to the Gartner Group “83% of data migrations fail outright or exceed their allotted budgets and implementation schedules”.

Data is a critical component to the success of any business. That data needs to be easily accessible and found in its rightful place, on the right system, and using the correct optimization.

To avoid all the aforementioned risks as much as possible, the testing performed on data migration ensures that data is kept intact and readily available.

Overall, in order to make data migration project a success, a combination of a stringent approach, proactive risk mitigation techniques and comprehensive test activities are inevitable.

Data Migration – Plan, Strategy, Process and more

Rahul — Mon, 10 Aug 2020 10:31:18 +0000

Businesses are spending billions of dollars in migrating data in migrating systems, yet many of the new systems fail to meet expectations.

In fact More than 50% of data migration projects will exceed budget and/or failing the business due to flawed strategy and execution, according to Gartner. Nevertheless, data migrations are a necessity and fact of IT life.

So let’s understand this complex though inevitable process at a glance.

What's Data Migration!

Data Migration is a process that includes moving data between formats, computer systems, and storage types.

We can also say that it is the process of selecting, preparing, extracting, and transforming data and permanently transferring it from one computer storage system to another.

Additionally, the validation of migrated data for completeness and the decommissioning of legacy data storage are considered part of the entire data migration process.

Why Data Migration!

Upgrading Computer / Legacy systems
Changing Computer / Legacy systems
Merging systems i.e. Data Integration

There are many reasons your enterprise might need to undertake a data migration project. For example, you might be replacing servers or storage devices or consolidating or decommissioning data center.

Data migration is also an essential step in the overall process of migrating on-premises IT infrastructure to a cloud computing environment.

Whether you’re moving to a public cloud, private cloud, hybrid cloud, or multi cloud environment, you’ll need to find a secure, cost-effective, and efficient method of migrating your data to its new storage location.

Improved technology often means business is thriving and is improving alongside its competitors. That said, the act of migrating data can leave previous information vulnerable. If critical data is lost, the business could be left at a standstill.

And What’s not the Data Migration!

Note that data migration is not the same thing as data conversion or data integration:

Data Migration

Data Conversion

Data Integration

Moving data between storage devices, locations, or systems. Includes subsets like quality assurance, cleansing, validation, and profiling.

Transforms data from a legacy application to an updated or new application. The process is ETL: extract, transform, load.

Combines stored data residing in different systems to create a unified view and global analytics.

6 Types of Data Migration

Data is stored on various media in files or databases, and is generated and consumed by software applications, which in turn support business processes. The need to transfer and convert data can be driven by multiple business requirements, and the approach taken to the migration depends on those requirements. The major migration categories are as follows:

Storage migration

Data center migration

Application migration

A business may choose to rationalise the physical media to take advantage of more efficient storage technologies.This will result in having to move physical blocks of data from one tape or disk to another, often using virtualisation techniques.

Migrating an entire data centre environment to the cloud or another location is a large-scale, comprehensive process. Completing such a migration project successfully—with minimal downtime or disruption to operations—requires careful planning and coordination.

Changing application vendor—for instance a new CRM or ERP platform—will inevitably involve substantial transformation as almost every application or suite operates on its own specific data model.

Database migration

Business process migration

Data migration to the cloud

It may be necessary to move from one database vendor to another, or to upgrade the version of database software being used.
Some modern applications are written to be almost entirely agnostic to the database technology, so a change from Sybase or SQL Server to Oracle should only require a testing cycle.
Database migration is an example of specialised workload migration.
Many public and private cloud providers offer tools that can facilitate or automate parts of the database migration process to ensure that your database remains secure throughout the transfer and that no data loss or corruption occurs.

Business processes operate through a combination of human and application systems actions.
When these actions change they require the movement of database or application data.
Examples of such migration drivers are mergers and acquisitions, business optimisation, and reorganisation to enter new markets or respond to competitive threat.
The last two categories directly affect the operational users of processes and applications.
A highly adaptive approach, a business-oriented audit and clear visibility for stakeholders are the key requirements in such migrations.

Today, businesses generate ever-growing amounts of data and face increasingly urgent pressure to maximise the value they extract from it.
Therefore, many organisations are choosing to move workloads to the cloud for a cost-effective and best-performing IT environment.
There are several options for transferring data from a local data centre to the cloud, broadly, they fall into following two categories.

1. Online migration, in which data moves across the Internet or a private or dedicated WAN connection.

2. Offline migration, in which data is transferred via a storage appliance that’s physically shipped between data centre and the target cloud storage location.

9 Steps for Successful Data Migration

1. Design a Strategy

Selecting a data migration strategy depends on the need for data migration. Is it because the organization is consolidating systems following an acquisition or a merger or is it because of data overload?
The design phase is where organisations define the type of migration — big bang or trickle. This also involves drawing out the technical architecture of the solution and detailing the migration processes.
Then you needs to decide whether to go for data migration services or one of the ETL tools to facilitate data migration.
These tools will identify the source and destination systems, generate workflows, and migrate the content. Alternatively, a code-free data migration solution can give you the freedom to carry out the migration at your convenience, speeds up the project, and reduces cost and risks.

2. Assess the Data source and Analyse

Before migrating data, you must know what you’re migrating, as well as how it fits within the target system.

So the next step is to analyse enterprise data. Answering the following questions can help with this assessment:

How much data is pulling over?
What that data looks like?
How much of the enterprise data requires migration?
Is the data structured or unstructured?
Is data redundancy going to be an issue?
What needs to migrate over, what can be left behind
How much of the data is ROT (redundant, obsolete, or trivial)?
Is the information old or recent?

3. Build a Migration Solution

Since, data migration is a one-time and large activity, it’s crucial to get it right.

A common tactic is to break the data into subsets and build out one category at a time, followed by a test. For particularly large migration projects, it might make sense to build and test in parallel.

4. Collect and Cleanse Data

This step involves removing data which is not required, compressing the remaining content, and converting data into the desired format.

Manual execution of these tasks can be time and resource intensive for large datasets. Therefore, automating these tasks can make the migration process more efficient.

5. Sort Data

Once you have profiled the data into a high quality and usable format, the next step is to categorise it according to the migration requirements.

It makes easier to route data into the right buckets. For example in Banking, data can be organised based upon the customer types, product type, portfolio types, account types and any other criteria.

6. Validate Data

This step involves reviewing the execution process. Assess the data rules and check if they are working the way they are supposed to, and watch for any exceptions in your dataflow.

After sorting the data, apply the data quality rules to determine the errors based on defined rules and to identify and eliminate the records in which the field ‘Quantity’ has a zero value.

7. Migrate

This is the final step in which everything falls into place. The steps mentioned above will give you well-organised and clean datasets.

Now all we have to do is migrate them from one system to another. Once the data is migrated to the target system, which can be a database or a data repository.

8. Conduct a Live Test

The testing process isn’t over after testing the code during the build phase.

It’s important to test the data migration design with real data to ensure the accuracy of the implementation and completeness of the application.

9. Audit

Once the implementation has gone live, set up a system to audit the data in order to ensure the accuracy of the migration.

Data Migration Strategies

“Big Bang” Migration

“Trickle” Migration

In a big bang data migration, as the name suggests, full transfer is completed within a limited window of time.

Live systems experience downtime while data goes through ETL processing and transitions to the new database.

It all happens in one time-boxed event, requiring relatively little time to complete. The pressure can be intense, as the business operates with one of its resources offline. This risks a compromised implementation.

Trickle migrations, in contrast, complete the migration process in phases.

During implementation, the old system and the new are run in parallel, which eliminates downtime or operational interruptions.

Compared to the big bang approach, trickle implementation can be fairly complex in design. However, usually reduces risks.

Advantages of big bang migration

Advantages of trickle migration

Less costly
Usually less complex
All changes happen one time only, in a relatively short space of time.

Less prone to expensive surprises.
Zero downtime required – as the migration is incremental
Easy rollback – If a single phase fails, it’s only that phase that needs to be rolled back and repeated

Disadvantages of a big bang

Disadvantages of a trickle

A high risk of expensive failure as problems may be discovered after complete migration.
If the migration fails, a complete roll back is required.
Requires downtime.

More expensive.
It requires to maintain multiple live environments

Risks and challenges

1. Data Loss

2. Security

3. Compatibility

During the data migration process, data loss can occur.

On a small scale, this may not be a problem as there is less data or easy to restore the missing data or backup. However this is a major challenge in large scale migration programs.

Organisations need to make sure that all data is securely encrypted before migration.

For offline migrations that involve shipping data storage devices, it’s important to verify security of the shipper’s freight and logistics services.

Compatibility issues are also possible in data transfer, such as changed operating systems and unexpected file formats.

Although the data is not actually lost, however, business cannot access it in the target system.

4. Long transfer times

5. Unexpected Higher Costs

It can be challenging to predict exact online transfer times.

Network impediments may restrict connection speeds or hardware restrictions might throttle the amount of data.

Unexpected costs often result from improper planning. For example, delays in online transfers additional charges.

Also, Keeping Vendor-provided storage appliances for longer than the agreed times can also raise the expanses.

Best Practices

1. Understand the data usage

2. Assess the environments

Who uses the data,
Who will use it in the future,
How will it be used?
Formatting requirements ?
Be sure to gather the requirements from relevant stakeholders and business units throughout the migration process.

Will the same operating system be running in source and target environments?
Will database schema or other formatting need to change?
Are there any issues that need to be addressed pre-migration?

3. Verify business requirements

4. Back up the data

What kind of migration timeline is necessary?
If a data centre is being decommissioned, when will its lease expire?
What types of data security must bee maintained throughout the migration process?
Is any data loss or corruption tolerable, and if so, how much?
How would unexpected delays affect the business?

Make sure there are backup resources and that they’ve been tested before proceeding.
One of the good practice is to create backup images that can be immediately restored to the original system.

5. Stick to the strategy

6. Test, test, test

The migration process can be complicated at times, so be ready for it and stick to the plan.

Make sure everything’s where it should be, create automatic retention policy, clean up the data and double check permissions.
During the planning and design phases, and throughout implementation and maintenance, test the data migration to make sure to achieve the desired outcome.

Data Migration Tools

On-premise data migration tools

Open source data migration tools

Cloud-based data migration tools

On-premise solutions are designed to migrate data between two or more servers or databases within a large or medium enterprise/network, without moving data to the cloud.
These solutions are optimal if you are performing tasks like changing data warehouses or moving the location of your primary data store.
Some companies prefer on-premise solutions due to security restrictions.

Open source is software you can use, modify, and share because its design is publicly accessible.
Typically, open source solutions are free or lower in cost than commercial alternatives.
Open source data migration tools can be a practical option for migrating your data if the project is not too large or complex.

Cloud-based data migration solutions are the latest generation and are designed to move data to the cloud.
Cloud-based solutions are optimal if you are already storing your data in the cloud or if you intend to move your data to the cloud.
Many organisations are now going for cloud based solutions considering cost efficiencies and enhanced security in moving data from on-premise to the cloud.
Moreover, cloud-based data migration tools tend to be flexible about the types of data they can handle.

Today, there are plenty of tools to facilitate enterprise data migrations. Generally, Data migration strategy determines which tools work best for the specific project.

Some popular Data Migration tools include the following:

• Rsync for host-based file-level migration
• Veeam to speed and simplify the process of migrating VM-based workloads.
• EMC Rainfinity for network-based file-level migration.
• Zerto offers a converged disaster recovery, backup, and workload mobility platform.
• HDS Universal Replicator for array-based block-level migration
• Rclone, an open source command line utility, used to migrate data to and from cloud object storage.
• Brocade DMM for network-based block-level migration
• Cyberduck is an open source FTP and SFTP client, used to transfer individual files or file volumes.
• Volume managers for host-based block-level migration
• Windows 10 Migration Accelerator
• Zinstall
• Acronis True Image
• Minitool Partition Wizard Free
• Laplink PCmover
• Samsung Data Migration Tool
• AOMEI Backupper Standard
• EaseUS Todo Backup/PC Trans Free
• Paragon Drive Copy Professional
• Samsung Smart Switch

Conclusion

Whatever the reason for the data migration, its ultimate aim should be to improve corporate performance and deliver competitive advantage.

To succeed, data migrations must be given the attention they deserve, rather than simply being considered part of a larger underlying project.

Lacking this and without proper planning, there is a high risk that the project will go over budget, exceed the allotted time, or even fail completely.

Following a structured methodology will reduce the pain of managing a complex data migration, whereas the correct choice of technology will lead to a successful outcome.

AWS Tutorial | Compute | EC2 | Launching and Terminating an Instance

Rahul — Sun, 12 Jul 2020 05:48:32 +0000

What is Amazon EC2?

Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) cloud.
Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster.
You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage.
Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.

Features of Amazon EC2

Amazon EC2 provides the following features:

Virtual computing environments, known as instances.
Various configurations of CPU, memory, storage, and networking capacity for your instances, known as instance types.
Preconfigured templates for instances, known as Amazon Machine Images (AMIs)
Multiple physical locations for your resources, such as instances and Amazon EBS volumes, known as Regions and Availability Zones.
Secure login information for instances using key pairs.
Persistent storage volumes for your data using Amazon Elastic Block Store (Amazon EBS), known as Amazon EBS volumes
A firewall that enables you to specify the protocols, ports, and source IP ranges that can reach your instances using security groups
Static IPv4 addresses for dynamic cloud computing, known as Elastic IP addresses
Metadata, known as tags, that you can create and assign to your Amazon EC2 resources

Lets get started with EC2 - Launch an EC2 instance

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

2. From the console dashboard, choose Launch Instance. Or go to Running Instances and then Launch Instances.

3. The Choose an Amazon Machine Image (AMI)page displays a list of basic configurations, called Amazon Machine Images (AMIs), that serve as templates for your instance. Select an HVM version of Amazon Linux 2. Notice that these AMIs are marked “Free tier eligible.”

4. On the Choose an Instance Type page, you can select the hardware configuration of your instance. Select the micro type, which is selected by default. Notice that this instance type is eligible for the free tier.

5. Choose Review and Launch to let the wizard complete the other configuration settings for you. or move step by step.

6. On the Review Instance Launch page, choose Launch.

7. On the Review Instance Launch page, under Security Groups, you’ll see that the wizard created and selected a security group for you. You can use this security group, or alternatively you can select the security group that you created when getting set up using the following steps:

Choose Edit security groups.
On the Configure Security Group page, ensure that Select an existing security group is selected.
Select your security group from the list of existing security groups, and then choose Review and Launch.

8. When prompted for a key pair, select Choose an existing key pair, then select the key pair that you created when getting set up.

Alternatively, you can create a new key pair. Select Create a new key pair, enter a name for the key pair, and then choose Download Key Pair. This is the only chance for you to save the private key file, so be sure to download it. Save the private key file in a safe place. You’ll need to provide the name of your key pair when you launch an instance and the corresponding private key each time you connect to the instance.

9. A confirmation page lets you know that your instance is launching. Choose View Instances to close the confirmation page and return to the console.

10. On the Instances screen, you can view the status of the launch. It takes a short time for an instance to launch. When you launch an instance, its initial state is pending. After the instance starts, its state changes to running and it receives a public DNS name. (If the Public DNS (IPv4) column is hidden, choose Show/Hide Columns (the gear-shaped icon) in the top right corner of the page and then select Public DNS (IPv4).)

11. It can take a few minutes for the instance to be ready so that you can connect to it. Check that your instance has passed its status checks; you can view this information in the Status Checks

How to Create a key pair before launching an EC2 Instance

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
In the navigation pane, choose Key Pairs.
Choose Create key pair.
For Name, enter a descriptive name for the key pair. Amazon EC2 associates the public key with the name that you specify as the key name. A key name can include up to 255 ASCII characters. It can’t include leading or trailing spaces.
For File format, choose the format in which to save the private key. To save the private key in a format that can be used with OpenSSH, choose pem. To save the private key in a format that can be used with PuTTY, choose ppk.
Choose Create key pair.

How to Create a security group

Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
From the navigation bar, select a Region for the security group. Security groups are specific to a Region, so you should select the same Region in which you created your key pair.
In the navigation pane, choose Security Groups.
Choose Create security group.
In the Basic detailssection, do the following:
1. Enter a name for the new security group and a description. Use a name that is easy for you to remember, such as your user name, followed by _SG_, plus the Region name. For example, me_SG_uswest2.
2. In the VPClist, select your default VPC for the Region.
In the Inbound rulessection, create the following rules (choose Add rule for each new rule):
- Choose HTTPfrom the Type list, and make sure that Source is set to Anywhere (0.0.0/0).
- Choose HTTPSfrom the Type list, and make sure that Source is set to Anywhere (0.0.0/0).
- Choose SSHfrom the Type In the Source box, choose My IP to automatically populate the field with the public IPv4 address of your local computer. Alternatively, choose Custom and specify the public IPv4 address of your computer or network in CIDR notation. To specify an individual IP address in CIDR notation, add the routing suffix /32, for example, 203.0.113.25/32. If your company allocates addresses from a range, specify the entire range, such as 203.0.113.0/24.

How to Terminate an Instance

In the navigation pane, choose Instances. In the list of instances, select the instance.
Choose Actions, Instance State, Terminate.
Choose Yes, Terminatewhen prompted for confirmation.

Amazon EC2 shuts down and terminates your instance. After your instance is terminated, it remains visible on the console for a short while, and then the entry is deleted.

11 Cross Browser Testing Tools

Rahul — Sat, 27 Jun 2020 12:13:36 +0000

We all are aware that Cross Browser Testing is an essential step in the website creation process.

It’s a type of non-functional testing which let you check whether your website and web-app works as intended across the popular and standard web browsers.

Different Browser-OS combinations i.e., on popular browsers like Firefox, Chrome, Edge, Safari—on any of the popular operating systems like Windows, iOS and Android.

Different devices with different capabilities i.e., users can view and interact with your website on popular devices—smartphones, tablets, desktops and laptops etc.

#1. Browserstack

Overview

Features

BrowserStack is a Cross-browser testing platform that allows testing of mobile applications and websites.
The product’s value proposition is to speed up release cycles and deploy websites and mobile applications working perfectly on all user interfaces.

Free Version – Yes
Free Trial – Yes
Platforms – Cloud, SaaS, Web
Training – Webinars, Documentation
Support – Online
Supports Parallel Execution.

Pros

Cons

A true representation of how a website will look on a mobile device.
A good variety of devices and browsers.
Easy to access BrowserStack from anywhere through Internet access.
Easy to inspect the code and any errors and also test changes on the logged on devices.
Quick and simple troubleshooting.

At times, it can take a while to connect to a device.
The screen sizes for desktop browsers are a bit complicated to adjust to test responsiveness.
Disconnects unexpectedly sometimes.

#2. Browsershots

Free Online service that will quickly take screenshots of the site in a range of browsers and versions
Browsershots makes screenshots of web design in different operating systems and browsers.
It is a free open-source online web application providing developers a convenient way to test their website’s browser compatibility in one place.
When you submit your web address, it is added to the job queue.

Pros

Cons

Can test the website across all versions of all popular mobile and desktop browsers, Operating systems and screen sizes, with filters like Java, Flash, Color depth, and more.
Supports Linux, Windows and BSD browsers.
It also includes a number of browsers you’ve probably never heard of like SeaMonkey, idori, Konqueror, Iceweasel etc

Delay in displaying the result when several browsers are selected and many times it shows timeout error or failure.

#3. CrossBrowserTesting

Overview

Offers the power of visual automation, designed to easily capture & manage test results.
Launch a Screenshot test to quickly spot unwanted browser inconsistencies, across dozens of desktop & mobile devices.
Run Dozens of Browsers in Parallel for faster test coverage.
Rapid Responsive Testing on the largest variety of OS/Browsers.
Easily create browser lists to manage your audience’s most used browsers.
Schedule your screenshot test & get notifications.

Pros

Cons

Ease of testing site over all the different browsers and devices. It is simple to use.
Large array of browsers and devices.
The ability to run headless browser testing.
One useful distinctive feature – actually LIST OUT the differences between the screenshots and how the website (or app) is rendering. This makes it extremely easy to spot layout issues in certain browsers.

Some tests fail due to a timeout when Selenium is waiting for a response for a command.
The mobile browsers fail a lot so it makes it hard to rely on for mobile testing

#4. Browserling

Overview

Features

Live interactive sessions.
Latest browsers.
Capture, save, and share screenshots of your web pages in all browsers.
Change screen resolution and resize browsers.
Record videos of your browsing sessions.

Plans – DEVELOPER, TEAM and PRO plans
Free Version – Yes
Free Trial – Yes
Platforms available – Cloud, SaaS, Web
Training – In Person, Live Online, Documentation

Pros

Cons

An easy interface to test your application on all browser types and operating systems.
Let you interactively verify the UI and functionality of the pages in specific browsers.

Page load speed can be quite slow sometimes.
Free trial gets you added to a queue with everyone else causing delay in starting the tests.

#5. Lambda Test

Overview

Features

LambdaTest is a scalable cloud-based Cross Browser testing platform meant for website and web app testing.
A wide range of Windows and Mac operating systems, along with many legacy and latest browsers.
Also test your website or web app on latest mobile browsers with Android and iOS mobile operating systems.
Seamless Collaboration with CI/CD pipeline or defect tracking tools.
Integrated developer tools to help users to debug issues in live testing.

Starting Price – USD 10.00/month
Free Version – Yes
Free Trial – Yes
Platforms available – Cloud, SaaS, Web
Training – In Person, Live Online, Documentation
Support – 24/7 (Live Rep)

Pros

Cons

Ease of Use and interactive.
Up to date with latest version of browsers.
Full-page screenshots are a huge plus.
Screenshot scheduling feature is useful.
Reliable customer support.
Cost effective alternative of other expansive similar products so value for money.

Minor UI issues.
Limited amount of free testing – 100 minutes in free trial.

#6. Comparium

Overview

Features

Comapium is a testing tool for websites across different Web-browsers.
Built for businesses, it enables them to test the functionality of their Web sites on major web browsers like Mozilla Firefox, Google Chrome, Safari, and Opera.

Test for all Web Browsers. There are multiple browsers and operating systems on which you can work by using Comparium.
These browsers and operating systems include the likes of Microsoft Internet Explorer 8.0, Safari 11.0, Google Chrome 75.0, 74.0, 73.0, Mozilla Firefox 68.0, 67.0, and Microsoft Internet Explorer 11.0.
An infinite number of Website tests.
Email Reporting even Comparium can send offline reports of the results of the website testing very quickly.
Comparium has a great screenshot web testing tool that has been acknowledged by many experts in the field.

Pros

Cons

Services of the first version are free to the users.
Easy to use, website has a how to use section for new users.
A great Lightweight and portable web page surveillance and review software for both desktop and mobile phones.
It allows simulated runs on several operating systems including Windows 7, macOS and Linux in parallel.
Facilitates web page evaluation simultaneously on various browser platforms.
Finding Differences feature with detailed results is quite amazing.
Shortly after test completion, the automatic screenshot is emailed to the designated recipient.

Long URL-formats is unsupported since only shorter URLs are approved.
Testing for Windows 10 and Android systems are not yet supported.

What does it cost?

Comparium has Guest and PRO plans.
Guest plan allows to use the following configurations: Win & Mac platforms w/o Dark Mode, latest Chrome and Firefox in 1024p width, completely free of charge.
The subscription for PRO plan costs $15/month.

#7. Sauce Labs

Overview

Features

Probably one of the most popular Cross Browser Testing Platform.
Cloud-based testing platform to offer the most up-to-date browser/OS combinations, mobile emulators & simulators, plus real device support.
Parallel tests allowed across many different browser and OS combinations.
Allows video recording of your entire test or flip through screen shots

Free Version – Yes
Free Trial – Yes
Platforms- Cloud, SaaS, Web
Training – In Person, Webinars, Documentation , Support, Online
Supports Parallel Execution – Yes

Pros

Cons

Sause labs User-Interface is very understandable, Easy to use and ‘rich’.
Offers an extensive test suite of mobile devices across platforms and version numbers.
Very easy to execute same test case in different devices at the same time.
Provides remote testing opportunities when devices are not available locally
Parallel Test Execution saves lot of time.
Video of test execution on each device.
With the “sauce labs cloud services” execute applications on various platforms with different browsers.
‘Record and play’ feature available.
Easy to use and intuitive dashboard.

Latency between input and action.
Many users complain about the speed of the tunnel.
Slow support.
High pricing as compared to the competitors.

#8. Multibrowser

MultiBrowser is a client side cross browser testing tool.
It runs natively on Windows, macOS and Linux and can be used to test for all major desktop and mobile browsers.
You can use it directly on your local machine and in your local network.

Pros

Cons

Real web browsers. Real testing.
Mobile Browser Emulators.
Record your testing session with the click of a button.
Get MultiBrowser now and test as many platforms as you want, including custom devices.
Built-in Video Recorder.
Brief documentation.

Latency in starting and running the tests.
Intermittent errors while running the tests.

#9. Testingbot

Overview

Features

TestingBot provides cross browser testing in the cloud.
By providing a browser grid of over 700+ browser combinations, you can test your website automatically and manually.
TestingBot provides an online cloud of +2300 browsers and devices.

Starting Price – USD 20.00/month
Free Version – Yes
Free Trial – Yes, 14 Days
Platform – Cloud, SaaS, Web
Supports Parallel Execution – Yes

Pros

Cons

Easy to use, very intuitive and well organised.
Run Automated and Manual tests on the browsers and devices.
Record and replay tests on all platforms.
Allows testing both manually and automatically in several browsers and platforms.
Offers Real Mobile Devices in its datacenter, capable of running both automated and manual tests.

Slow connection when doing manual testing.
Uses mobile emulators instead of real devices

#10. Ghostlab

Overview

Features

Ghostlab is automated testing software, and includes features such as parameterized testing, Requirements-Based testing, and test script reviews.
An on-premise solution that helps inspect DOM and CSS through synchronized testing

Starting Price – USD 49.00 one time
Free Version – No
Free Trial – Yes
Platform – Windows, Mac
Supports – Online
Parameterized Testing supported – Yes

Pros

Cons

The ability to set up site with Ghostlab, fire up a QR code, scan it on own devices and begin testing.
Automatically tests clicks, page refreshes, scrolls, and form inputs on all synchronized devices and browsers.
All connected devices and browsers are kept in sync.
Allows test on a variety website on various browsers and mobile devices.
Users can use Ghostlab’s built-in inspector to fix CSS bug while testing.
Can take a screenshot directly from within Ghostlab.
Inspect DOM and CSS on any device.
Automagic form filling.
Testing page provides useful information such as pixel ratio and viewport dimensions.

Only available on Windows and Mac computers; no capability for Linux devices.

#11. Perfecto

Overview

Features

Perfecto is the leading testing platform for web and mobile apps. It enables exceptional digital experiences and helps the customers strengthen every interaction for web and mobile apps through a cloud-based test environment.
It makes continuous testing achievable with features like BDD, codeless automation, parallel test executions & bursting capabilities, open integrations with leading DevOps tools and ML-driven test analytics.
Perfecto claims and focuses on following 4 aspects:
Coverage
Security
Scalability
Integration

Starting Price – USD
300.00 one time
Free Trial – Yes
Platform – Cloud, SaaS, Web, Mobile
Training- In Person, Live Online, Documentation
Support – Online, 24*7
Parameterized Testing supported – Yes

Pros

Cons

Large inventory of device
Accelerated Executions Makes web app testing over 50% faster with Perfecto than other solutions.
Quick and easy Test failure analysis – built-in analytics solution provides detailed test reports, a CI dashboard, heatmaps, and root cause analysis.
CI/CD Integration.
Supports Hybrid objects approach, which consists of both Visual and native objects for iOS and Android apps.
In addition to Cross Browser testing, Perfecto contains full testing solution ; Users can execute Automation or interactively test more complex scenarios in the cloud.

Intermittent slowness in response as well as in product updates.
High license cost

Conclusion

The list of cross-browser testing tools is endless now a days, selection of a suitable cross-platform tool depends on your requirement as each browser compatibility checking tool comes with its own unique features, advantages and disadvantages.

And Before finishing.....

AWS Tutorial – 3 | Identity Access Management

Rahul — Fri, 22 May 2020 02:14:45 +0000

What is IAM?

AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.

Manage users and their level of access to the AWS console.
Set users, permissions and roles.
Create multiple users, each with its own security credentials, controlled and billed to a single AWS account.
Create and manage policies to grant access to AWS services and resources.
Assign and control security credentials to each user.
Create roles for your AWS account
Create and manage IAM groups.

Features of IAM

Centralised control of your AWS account: You can control creation, rotation, and cancellation of each user’s security credentials. You can also control what data in the aws system users can access and how they can access.
Enhanced security: IAM enables security best practices by allowing you to grant unique security credentials to users and groups to specify which AWS service APIs and resources they can access.
Shared Access to your AWS account: Users can share the resources for the collaborative projects.
Granular permissions: It is used to set a permission that user can use a particular service but not other services.
Multifactor Authentication: An AWS provides multifactor authentication as we need to enter the username, password, and security check code to log in to the AWS Management Console.
Provide temporary access for users/devices and services where necessary: If you are using a mobile app and storing the data in AWS account, you can do this only when you are using temporary access.
Integrates with many different aws services: IAM is integrated with many different aws services.
Eventually Consistent: IAM service is eventually consistent as it achieves high availability by replicating the data across multiple servers within the Amazon’s data center around the world.
Free to use: AWS IAM is a feature of AWS account which is offered at no additional charge. You will be charged only when you access other AWS services by using IAM user.

What are IAM users?

An IAM user is an entity that you create in AWS.
The IAM user represents the person or service who uses the IAM user to interact with AWS.
A primary use for IAM users is to give people the ability to sign in to the AWS Management Console for interactive tasks and to make programmatic requests to AWS services using the API or CLI.
A user in AWS consists of a name, a password to sign into the AWS Management Console, and up to two access keys that can be used with the API or CLI.
When you create an IAM user, you grant it permissions by making it a member of a group that has appropriate permission policies attached or by directly attaching policies to the user.

What are IAM Groups?

An IAM group is a collection of IAM users.
Add users to or remove them from a group.
A user can belong to multiple groups.
Groups cannot belong to other groups.
You can use groups to specify permissions for a collection of users, which can make those permissions easier to manage for those users.
For example, you could have a group called Admins and give that group the types of permissions that administrators typically need.
Any user in that group automatically has the permissions that are assigned to the group.

What are IAM Roles?

An IAM role is very similar to a user, in that it is an identity with permission policies that determine what the identity can and cannot do in AWS. However, a role does not have any credentials (password or access keys) associated with it.
Instead of being uniquely associated with one person, a role can be taken by anyone who needs it. An IAM user can assume a role to temporarily take on different permissions for a specific task.

What problems do IAM roles solve?

How many IAM roles can be created?

Through IAM roles, you can delegate access with defined permissions to trusted entities without sharing long-term access keys.
You can use IAM roles to delegate access to IAM users managed within your account, to IAM users under a different AWS account, or to an AWS service such as EC2.

Users can create up to 1,000 IAM roles under one AWS account. For further roles, need to contact Amazon support team through the Limit increase form.

IAM Users, Groups and Role at a glance

IAM User

IAM Group

IAM Role

An IAM user has permanent long-term credentials and is used to directly interact with AWS services.

An IAM group is primarily a management convenience to manage the same set of permissions for a set of IAM users.

An IAM role is an AWS Identity and Access Management entity with permissions to make AWS service requests.
IAM roles cannot make direct requests to AWS services; they are meant to be assumed by authorized entities, such as IAM users, applications, or AWS services such as EC2.

Policies and Permissions

A policy is an object in AWS that, when associated with an identity or resource, defines their permissions.
You manage access in AWS by creating policies and attaching them to IAM identities (users, groups of users, or roles) or AWS resources. A
AWS evaluates these policies when an IAM principal (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents.

Six Policy Types

IAM policies define permissions for an action regardless of the method that you use to perform the operation.
For example, if a policy allows the GetUser action, then a user with that policy can get user information from the AWS Management Console, the AWS CLI, or the AWS API.
When you create an IAM user, you can choose to allow console or programmatic access.
If console access is allowed, the IAM user can sign in to the console using a user name and password.
Or if programmatic access is allowed, the user can use access keys to work with the CLI or API.

Identity-based policies

Resource-based policies

Permissions boundaries

Attach managed and inline policies to IAM identities (users, groups to which users belong, or roles). Identity-based policies grant permissions to an identity.

Attach inline policies to resources.
The most common examples of resource-based policies are Amazon S3 bucket policies and IAM role trust policies.
Resource-based policies grant permissions to the principal that is specified in the policy.
Principals can be in the same account as the resource or in other accounts.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Organizations SCPs

Access control lists (ACLs)

Session policies

Use an AWS Organizations service control policy (SCP) to define the maximum permissions for account members of an organization or organizational unit (OU).
SCPs limit permissions that identity-based policies or resource-based policies grant to entities (users or roles) within the account, but do not grant permissions.

Use ACLs to control which principals in other accounts can access the resource to which the ACL is attached.
ACLs are similar to resource-based policies, although they are the only policy type that does not use the JSON policy document structure.
ACLs are cross-account permissions policies that grant permissions to the specified principal.
ACLs cannot grant permissions to entities within the same account.

Pass advanced session policies when you use the AWS CLI or AWS API to assume a role or a federated user.
Session policies limit the permissions that the role or user’s identity-based policies grant to the session.
Session policies limit permissions for a created session, but do not grant permissions.

More on Policy and Permission

What is a policy summary?

How do permissions work?

How do I assign permissions using a policy?

What does a policy look like?

If you are using the IAM console and choose a policy, you will see a policy summary.
A policy summary lists the access level, resources, and conditions for each service defined in a policy (see the following screenshot for an example).
The access level (View, Read, Write, or Permissions management) is defined by actions granted for each service in the policy.
You can view the policy in JSON by choosing the JSON button.

Access control policies are attached to users, groups, and roles to assign permissions to AWS resources.
By default, IAM users, groups, and roles have no permissions; users with sufficient permissions must use a policy to grant the desired permissions.

To set permissions, you can create and attach policies using the AWS Management Console, the IAM API, or the AWS CLI.
Users who have been granted the necessary permissions can create policies and assign them to IAM users, groups, and roles.

The following policy grants access to add, update, and delete objects from a specific folder, example_folder, in a specific bucket, example_bucket.

{
“Version”:”2012-10-17″,
“Statement”:[
   {
     “Effect”:”Allow”,
     “Action”:[
   “s3:PutObject”,
   “s3:GetObject”,
   “s3:GetObjectVersion”,
   “s3:DeleteObject”,
   “s3:DeleteObjectVersion”
     ],
“Resource”:”arn:aws:s3:::example_bucket/example_folder/*”
}
]
}

Next--> AWS Databases

AWS Tutorial – 2 |Getting Started with AWS | Free AWS Signup

Rahul — Sat, 16 May 2020 12:15:33 +0000

1. To create an AWS account, open this link https://aws.amazon.com

Note: When you create an AWS account you have 12 months of Free Tier Access. You can read more about this offer at https:aws.amazon.com/free As an example in this account you would have 5Gb of Standard Storage, 750 hours of EC2 and RDS, I million calls of AWS Lambda and many more for 12 months.

2. Select the Type of Account you want create, either Professional or Personal. In this example I’m creating a Personal account.

3. Use a valid credit card Fill in the required Payment Information fields and then click on ‘Secure Submit’

4. Setting up the MFA:

And finally you land up on the Console page.

Next --> AWS Identity Access Management(IAM)

AWS Tutorial – 1 | Cloud Fundamentals | 30,000-Foot view of AWS

Rahul — Sun, 26 Apr 2020 10:49:29 +0000

AWS (Amazon Web Service) is a global market leader in Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) industry, offering a wide range of remote cloud services. In this article,

It’s a cloud computing platform that enables users to access on demand computing services like database storage, virtual cloud server. Amazon’s cloud web hosting platform offers flexible, reliable, scalable, easy-to-use, and cost-effective solutions. This tutorial covers various important topics illustrating how AWS works and how it is beneficial to run your website on Amazon Web Services.

So first let's understand what is Cloud Computing?

If you’re unsure about the Cloud Computing, you are probably among the people already using cloud services, like online banking and social networks, but don’t realize it.

The “cloud” is a set of different types of hardware and software that work collectively to deliver many aspects of computing to the end-user as an online service.

Cloud Computing is the use of hardware and software to deliver a service, usually over the Internet. With cloud computing, users can access files and use applications from any device that can access the Internet.

Cloud computing is also a term referred to storing and accessing data over the internet. In cloud computing, you can access data from a remote server.

One of the most popular example of a Cloud Computing provider is Gmail. Gmail users can access files and applications hosted by Google via the internet from any device.

What are Public, Private and Hybrid Clouds?

Public cloud

Private cloud

Hybrid cloud

· A Public Cloud is based on the standard cloud computing framework which consists of files, applications, storage and services available to the public via the internet. Gmail is an example of a Public Cloud.

· Public clouds are owned and operated by a third-party cloud service providers, which deliver their computing resources, like servers and storage, over the Internet.

· With a public cloud, all hardware, software, and other supporting infrastructure is owned and managed by the cloud provider. You access these services and manage your account using a web browser.

A Private Cloud is comprised of files, applications, storage and services that are implemented and protected within a corporate firewall, under the control of a corporate IT department.
A private cloud refers to cloud computing resources used exclusively by a single business or organization.
A private cloud can be physically located on the company’s on-site data center.
A private cloud is one in which the services and infrastructure are maintained on a private network.
An example of a Private Cloud would be a company that uses Microsoft Exchange because Microsoft Exchange can only be accessed by an authorised user through a secure VPN connection.

Hybrid clouds combine public and private clouds, bound together by technology that allows data and applications to be shared between them.
By allowing data and applications to move between private and public clouds, a hybrid cloud gives your business greater flexibility, more deployment options, and helps optimize your existing infrastructure, security, and compliance.

Three models of cloud service

Infrastructure as a service (IaaS)

Platform as a Service(PaaS)

Software as a service (SaaS)

Infrastructure-as-a-service solutions generally get to the level of ‘bare metal’ — the physical servers, networking components, and device storage necessary to make platforms (and, hence, services) functional.
IaaS is popular with corporate clients, with tradeoffs between speed, cost, and privacy that each vendor balances in different ways.
The most basic category of cloud computing services. With IaaS, you rent IT infrastructure—servers and virtual machines (VMs), storage, networks, operating systems—from a cloud provider on a pay-as-you-go basis

Vendors that offer platform-as-a-service portfolios generally face corporate clients.
PaaS products include virtual servers, operating environments, database environments, and any other middle-ware component that sits between the hardware and the consumer-facing application.
PaaS is designed to make it easier for developers to quickly create web or mobile apps, without worrying about setting up or managing the underlying infrastructure of servers, storage, network, and databases needed for development.

Cloud services using a software-as-a-service model, or SaaS, provide fully functional programs to end users even though the programs may not be resident on their local computers.
Software as a service is a method for delivering software applications over the Internet, on demand and typically on a subscription basis.
With SaaS, cloud providers host and manage the software application and underlying infrastructure, and handle any maintenance, like software upgrades and security patching.
Email providers like Gmail and Outlook.com are SaaS applications, as well as just about any computer program that runs inside a browser.

Now let's see what's AWS?

AWS (Amazon Web Service) is a comprehensive cloud computing platform offered by Amazon that offers flexible, reliable, scalable, easy-to-use and cost-effective cloud computing solutions and enables user to access on-demand computing services like database storage, virtual cloud server, etc.
It works on “pay-as-you-go” model that means – you don’t need to pay upfront for services it offer. You pay for only how much service (mostly bandwidth) you consumed at the end of billing cycle.
The platform is developed with a combination of infrastructure as a service (IaaS), platform as a service (PaaS) and packaged software as a service (SaaS) offerings.

AWS Components

AWS Global Infrastructure

AWS Regions

Availability zone

Edge Locations

AWS Region, is kind of framework, in which we have all the available providers.

An AWS Region is a geographical location with a collection of availability zones mapped to physical data centres in that region. Every region is physically isolated from and independent of every other region in terms of location, power, water supply.

Example:For EC2 instance, Storage, DB – we need a region in which we can build our services). We have 17 Regions at the time of preparing this tutorial.

An availability zone is a facility that can be somewhere in a country or in a city, it’s a logical data center in a region available for use by AWS customers.

Each zone in a region has redundant and separate power, networking and connectivity to reduce the likelihood of two zones failing simultaneously.

An availability zone can be a several data centres, but if they are close together, they are counted as 1 availability zone.

Edge Location are the locations where end user services are provided.

Edge locations are the endpoints for AWS used for caching content.

Edge locations are more than regions. Currently, there are over 150 edge locations.

Next --> AWS Identity Access Management(IAM)

20 API Testing Tools – Beyond SOAP UI and Postman

Rahul — Fri, 10 Apr 2020 01:36:06 +0000

We all know about API testing well. In my previous article – API testing, Key Terminologies and more……we learnt about the basics of API testing. Let’s now take a look on some of the API Testing tools.

To learn something new, I haven’t considered the leaders like SOAP UI, Postman, JMeter. Rather I have explored some tools which may be lesser known but have more or less same features and give a solid performance therefore worth giving a shot.

#1 ACCELQ

Zero code API Test Automation on Cloud.
Support for leading technologies.
API and UI Test Automation in the same simplified flow.
CI Driven regression suite executions.
Dynamic Environment management.
Directly correlate Business process and corresponding API for complete coverage.
API Test Case Management, Test Planning, Execution and tracking governance.
Trial – 14 days free trial

#2 API Fortress

API Fortress is a REST & SOAP API, Web Services, and Microservices testing platform.
Out-of-the-Box Integrations.
Deploy On-Premises or in the Cloud.
Quickly Generate and Edit Functional Tests
Detailed Reporting.
30 days free trial

#3 API Science

API Science is an advanced API Monitoring service.
Monitor own APIs
Multi-step REST transactions
Advanced tests & validations
Get alerted on any issue
Run tests at any time
Monitor your own API & 3rd party APIs
Customized reporting
Intelligent API diagnostics
Collaborate using
team plans

#4 Apigee

Easily create API proxies and visually configure or code API policies as steps in the API flow.
Enforce consistent security best practices and governance policies across all APIs.
Publish APIs by providing reference documentation, manage the audience for an API from portal.
Analyze APIs, API traffic data, investigate traffic spikes, and trace live API calls.
Monitor APIs to ensure API availability.
Publish APIs to a customizable portal, so that developers can easily explore, test, get API keys, and innovate fast.
60 days free trial.

#5 Assertible

Can automatically sync API tests with the latest changes in the specification.
Integrates with the tools e.g. push code to GitHub or send alerts to Slack if failures happen.
Easily create tests for web service.
Continuously test multiple deployment and environment.
Set up scheduled monitoring to continuously test the services.
Run the same tests on multiple environments.
Automatically execute post-deploy smoke tests.
HTTP assertions allow you to create domain specific test.

#6 Citrus Framework

Citrus is open source and licensed under Apache License 2.0.
Test automation by creating repeatable integration tests within continuous build.
Test complex messaging scenarios with step by step sequence of operations.
Use advanced logic in test cases including loops, delays, retries, parallel sections.
Json message payload validation.
JsonPath element validation.
Groovy Json validation.
Simple test creation.
IDE support – execute tests as TestNG/JUnit tests from Eclipse, IntelliJ IDEA or NetBeans.

#7 HTTPMASTER

Development and test tool for REST web services and API applications.
Can display and validate the most common REST formats; XML, JSON, and HTML
Broad set of http properties.
Response data validation
Simulate variations of input data or create global values for reuse across multiple API calls.
Enables to define chaining items of different types.
Support for data upload which enables users to post plain text data, contents of file, or mix of plain text data and disk files.
Free 30-day trial

#8 Insomnia

Free and open source on Mac, Windows, and Linux.
Reuse API keys or session IDs. Define environment variables globally or switch between sub-environments.
Generate http code for over thirty language libraries, including Curl, NodeJS, Go, Swift, Python, Java, C.
Free version available.

#9 Mockbin

Mockbin allows you to generate custom endpoints to test, mock, and track HTTP requests & responses between libraries, sockets and APIs.
Mock custom endpoints using any HTTP Archive (HAR) response object.
Test using your preferred language.
Mockbin accepts all standard Methods and allows method overriding in addition to Get and Post.
Mockbin supports output in JSON, YAML and XML, as well as an HTML.
Log and inspect incoming calls to your custom endpoints.

#10 Ping API

Ping-API is API testing allows to write test script in JavaScript and CoffeeScript.
It allows inspecting HTTP API call with a complete request and response data.
Can schedule test in every minutes or hour
Support for writing script to set request headers, body and URL parameters.
It also supports validation of response headers and body.
Easy to generate test script with the input of API parameters.
Free for limited projects, varied pricing afterwards.

#11 Postwoman

Postwoman is an open-source API request builder. It helps you create your requests faster, saving you precious time on your development.

It also helps to create the requests faster, saving precious time on overall development.

Free, Open Sourced
Runs online
Self Hosted
Multi-platform & multi-device support
Progressive Web App
WebSocket testing
Customizations allowed
Pre-Request Scripts and environment variables setup
Docker support

#12 pyresttest

A REST testing and API microbenchmarking tool.
Tests are defined in basic YAML or JSON config files, no code needed.
Minimal dependencies (pycurl, pyyaml, optionally future), making it easy to deploy on-server for smoketests/healthchecks.
Supports generate/extract/validate mechanisms to create full test scenarios.
Returns exit codes on failure, to slot into automated configuration management/orchestration tools (also supplies parseable logs).
Logic is written and extensible in Python.
Return exit codes for failed results.
The building of test scenarios with generate/extract/validates mechanisms.
Easy deployment on the server.

#13 ReadyAPI

ReadyAPI is an integrated suite of API testing tools that includes:

SoapUI Pro (functional testing)
LoadUI Pro (performance testing)
ServiceV Pro (service virtualization)

Metrics & Data In One Place.
Create and maintain all the information relating to projects including test cases, data sources, and object components in one place for re-usability and management.
The entire ReadyAPI suite uses a central, standardized reporting function to support the measuring and displaying of data and metrics in real time.
Continuous Integration with ReadyAPI.
ReadyAPI provides many options for scripting to create functional, load, or security tests, using either Groovy or JavaScript.
With API Discovery, ReadyAPI will listen to live traffic of an API, recording responses and different interactions captured during the API activity.

#14 RestSharp

RestSharp is a popular HTTP client library for .NET.
Featuring automatic serialization and deserialization, request and response type detection, variety of authentications.
JSON, XML and custom serialization and deserialization.
Basic, OAuth 1, OAuth 2, JWT, NTLM are supported.
HTTP call can be customized.
Send objects as the request body in JSON or XML, or as a form.

#15 SOAPSonar

SOAPSonar provides automated testing of each API independent of client, device or Consumer.
Automatically generating a UI to provide immediate feedback to developers, testers.
Validates each API independently, shortens the critical path and supports Agile development.
Fully automated, requires no scripting or maintenance thus reducing time and skill-set required.
Integration with variety test management software hence provides detailed progress reports.
Regression testing can be automated, providing early identification of changes to the API or versions.
Support for message and protocol layer identity standards allows for cloud testing.

#16 SOAtest

Broadest support for over 100 message formats and protocols.
With visual drag-and-drop tooling, users can create the most complex test scenarios without having to write a single line of code.
Test flow logic, complex assertions, looping, data driving, and keyword association, such as BDD with Cucumber, can all be easily built with minimal technical experience.
With its SMART API Test Generator plugin for Chrome, SOAtest monitors activity in your web UI from manual or exploratory testing and converts the behind-the-scenes API calls into meaningful API test scenarios.
Integration with existing CI/CD systems (i.e. Jenkins, TeamCity, Bamboo, Azure DevOps).
Collects REST and SOAP API coverage as part of test execution, reporting the test case coverage, missing or incomplete.

#17 TestMace

Creating and testing complex scenarios quickly.
Creating tests without actual programming.
Powerful autocomplete feature and highlighting of current variables values, functions etc.
A well-organized project structure and human-readable file format, which allow to store your project in the version control system repo and review tests and scenarios.
Advanced JS support in the built-in code editor, including the autocomplete feature and static analyzer.

#18 vRest

Record test cases by manually using your application once.
vrunner lets you execute your test cases just by running a command on your terminal.
Integration with Continuous Integration servers like Jenkins, TeamCity, Visual Studio Team Services etc. into vREST.
Automatically log issues into bug tracking systems like JIRA, Bugzilla, etc and notify team using Slack.
vREST lets you write test cases to validate your web/mobile application.
Supports Data driven testing. vREST lets you write test cases to validate your web/mobile application.
Schedule your test cases using cronjob utility (in Mac or Linux) or Task Scheduler (in Windows).

#19 WebInject

WebInject is a free tool for automated testing of web applications and web services.
It can be used to test individual system components that have HTTP interfaces (JSP, ASP, CGI, PHP, AJAX, Servlets, HTML Forms, XML/SOAP Web Services, REST, etc).
Offers real-time results display and may also be used for monitoring system response times.
WebInject uses an XML API for defining and loading test cases. You can use WebInject without ever seeing its internal implementation.
Result reports are generated in HTML (for viewing) and XML

#20 Paw

Paw is a full-featured HTTP client that lets you test and describe the APIs you build or consume.
It has a native macOS interface to compose requests, inspect server responses, generate client code and export API definitions.
Easily craft your HTTP requests, try and archive them.
Setup HTTP Headers, URL parameters, form-encoded POST key-values or text.
Get easy-to-read responses with well formatted headers and body.
Sync your projects with your team through Paw for Teams.
Simple request chaining (right-click in the UI, no code or globals).
Great workflow for when writing a fast-changing API

Conclusion

This list can go on and on however we have to stop somewhere. The above list is an effort to explore some different API testing tools which may not be the most popular ones but work well and perform effectively.

Though all API test tools listed above work well and are great options.however, there is no one-size-fit-all tool. There is no perfect tool for all organisations: Every organization has different requirements.

So the key is – carefully considering the current and future project’s requirements and pros and cons of each solution.

API Testing, Key Terminologies and more…..

Rahul — Tue, 31 Mar 2020 23:27:34 +0000

An application programming interface (API) at its core is a formal specification that acts as a guaranteed contract between two separate pieces of software. We can also define API as an interface between two software applications by allowing them to communicate with each other.

APIs have become the focal point of programming improvement, interfacing and transferring information and rationale across frameworks and applications. Fortunately, testing them can enormously improve the productivity of your testing procedure overall, helping you convey programming quicker than at any other time.

So What is API Testing!

Applications often have three layers:

Data layer
Service (API) layer
Presentation (UI) layer

The API layer contains the business logic of an application – the rules of how users can interact with services, data, or functions of the app. Since the API or service layer directly touches both the data layer and the presentation layer, it presents the sweet spot of continuous testing for Testing and Development teams.

API testing involves testing the application programming interfaces (APIs) for functionality, security, Performance. Since APIs lack a GUI, API testing is performed at the Service layer.

Testing types in API Testing

1.Unit Testing: For testing the usefulness of individual activity.

2. UI Testing: For testing the usefulness of UI as a component of start to finish coordination tests to guarantee the UI capacities true to form.

3. Functional Testing: Functional testing is simply a test of specific functions within the code base. These functions represent specific scenarios (successful cases) and error handling.

Due to the nature of the test, we should expect a few stated responses. We should expect either an error (and thus, the appropriate error codes and handling instructions) or a corrected response that bears the material we’ve requested.

Functional testing should be done with the positive scenarios as well as both errata and edge cases.

4. Load Testing: For testing the Functionality and execution under Load conditions.

Load testing takes on a few different scenarios in order to ensure peak performance. The first of these scenarios is called the “baseline“, and tests the API against the theoretical regular traffic the API expects in normal, day-to-day usage.

A second load test is generally done with the theoretical maximum traffic. This is done to ensure that even during full load, methods are in place to safely throttle requests.

Finally, an overload test is performed to test the theoretical maximum and to add 10–20% additional traffic on the top. While this type of testing anticipates some sort of failure, it is more like a test of the API functionality to validate the error code generation and to ensure any failures are handled during high load.

5. Error Detection: For recognising any blunders, for example, special cases and asset spills.

This type of test is focused on the actual running of the API. Other tests are concerned with the result of implementing the API in a scenario, error detection is concerned with the universal results of utilizing the API codebase. These types of tests have one of the following focuses:

• Monitoring: Compiled code is tested for various implementation errors, handler failures to ensure there is no security issue in the code base.

• Execution Errors: The code should respond to valid requests in a predictable, known way, and should fail invalid requests.

• Resource Leaks: Invalid requests, purposefully overflowing commands, and other “illegal but common” types of requests are submitted to the API to test for memory, resource, data, or operational leaks and insecurities.

• Error Detection: The code is put through known failure scenarios to ensure that errors are identified and handled.

6. Usability Testing : It is easy to hit the APIs on different platforms and test the usability of a software.

7. Security Testing: Security testing also includes penetration testing, and fuzz testing.

Security Testing includes validation of encryption methodologies and validating the design of the access control through user rights management and validating authorisation checks.

In Penetration testing, API is attacked by someone with limited working knowledge of the API to assess the threat vector from outside. These attacks can be limited to specific functions, resources, or processes, or can target the entirety of the API.

In Fuzz testing, massive amounts of random data, known as “noise” or “fuzz,” is forced into the system in order to attempt a forced crash, overflow, or any other negative behaviour. This is done to test the API at its absolute limits, and serves somewhat as a “worst case scenario.”

8. Interoperability and WS Compliance testing: This sort of testing applies to SOAP APIs and checks for following two aspects:

Interoperability between SOAP APIs is checked by ensuring conformance to the Web Services Interoperability profiles. By conforming to these guidelines and utilising these tests, interoperability between SOAP APIs can be confirmed

Secondly, WS-* compliance is tested to ensure standards such as WS-Addressing, WS-Discovery, WS-Federation, WS-Policy, WS-Security, and WS-Trust are properly implemented and utilized.

9. Discovery Testing : API testing helps in identifying APIs where the resources are not exposed.

Infiltration Testing: For identifying any vulnerabilities of an application from aggressors.

API Testing Best Practices

1. Deciding What to Test and what not i.e Create test cases for all possible API input combinations to get complete test coverage

2. Use Realistic Data

3. Don’t Repeat Yourself.

4. Write clear tests that easily enable debugging.

5. Design the tests to run under different SUT configuration options.

6. Test Positive and Negative Outcomes

7. Use Data to Drive Dynamic Assertions

8. Track API Responses

9. Test the API for failures; until you get the output as failed.

10. Reuse test cases and monitor the API in production.

A simple API Testing process

Step 1: Create API testing requirements. To be able to plan API tests, we need to answer following questions:

We need to determine testing boundaries and requirements.
Purpose of the API and its target consumer;
Workflow of the application;
Features and functions of the API;
Other APIs this API interacts with;
Aspects of testing;
Problems we’re testing for;
Priorities in testing;
Desired output;
Definition of Pass and Fail.

Step 2: Establish the API test environment. After the functional scope, the next step is setting up an API test environment. It requires the configuration of the servers, databases, and every resource the API interacts with, depending on the project requirements.

Step 3: Make a trial API call. Before starting the testing, perform a Sanity test i.e. make a test API call just to check the API is operational.

Step 4: Define the input parameters. Plan all possible input combinations to verify the results to determine whether the API performs as expected.

Step 5: Create API test cases. After all the preparations are done, it’s the time to write and execute test cases followed by compare actual results with the expected ones. A good practice is to group them by test category. Examples of API test cases may include:

Positive and Negative test values to check the response is Passed or Failed.
When there is no return value in response.
Verification of response after data structure updates;
Validating resources modified by the API call.

Web Service / API Testing Terminologies

1. XML

XML stands for extensible Markup Language
It’s a mark-up language much like HTML
XML was designed to store and transport data
XML was designed to be self-descriptive
It has sender information.
It has receiver information
It has a heading
It has a message body.

2. WSDL

WSDL stands for Web Services Description Language
WSDLs are an XML format that tells you how to access a web service.
WSDL is used to describe web servicesWSDL is written in XML

3. SOAP

SOAP stands for Simple Object Access Protocol. It helps in exchanging structured information between computer networks.
SOAP allows communication between different operating systems using XML.

4. SOA

SOA (Service-oriented Architecture) is a way in which companies can organize software that can be quickly changed to respond to the requirements of the marketplace.
Web Services are small units of software that run in a network. They are typically written to handle a specific business process. Web services can be strung together in multiple ways and used by different applications to create the desired functionality.

5. REST

REST stands for Representational state transfer.
These are web services that provide interoperability between computer systems over the internet.
RESTful Web services provide a predefined set of stateless operations and allow requesting systems to access and manipulate textual representations of web resources.

6. SOAP vs REST

SOAP stands for Simple Object Access Protocol.
SOAP is a protocol. It defines some standards that should be followed strictly.
Web services following SOAP principles are called SOAP Web services
SOAP message request is processed slower as compared to REST.
SOAP supports only XML data format.
SOAP is not very easy to implement so it is preferred less.
SOAP requires more bandwidth and resources.
SOAP Message contains Envelope, body and header.
It does not use web caching mechanism.
SOAP is commonly used in payment gateways, financial and telecommunication services.

REST stands for Representational State Transfer.
REST is an architectural style.
Web services following REST Architectural style are called RESTful Web services
REST message request is processed faster as compared to SOAP.
REST supports data formats like plain text, XML, HTML, JSON, etc.
REST is easier to implement.
REST requires less bandwidth and resources.
REST Message is enclosed in HTTP and uses HTTP Method and URI (Uniform Resource Identifier)
It uses web caching mechanism i.e. data can be cached at the client.
REST is commonly used in social media, web chat and mobile services.

7. Web Services vs APIs

All web services are APIs.
It can only be hosted on IIS.
It is not open source but can be used by any client that understands XML.
It requires a SOAP protocol to receive and send data over the network, so it is not a light-weight architecture.
A Web service uses only three styles of use: SOAP, REST and XML-RPC for communication.
It only supports the HTTP protocol.

All APIs are not web services.
It can be hosted within an application or IIS.
It is open source and it can be used by any client that understands JSON or XML.
It is light-weight architecture and good for devices which have limited bandwidth, like mobile devices.
API may use any style of communication.
It supports the HTTP protocol: URL, Request/Response Headers, caching, versioning, content formats.

8. JSON

JSON stands for JavaScript Object Notation and was designed to be a lightweight data interchange format. JSON is definitely becoming more popular and is now replacing XML in certain situations for API data exchanging. The site www.json.org describes how JSON is built on two structures:
1. “A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array.
2. “An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.”

9. SOAP Message Elements

1) SOAP Envelope – The SOAP Envelope is always the top element in the message.

2) Header – the Header is optional and is the first child element to appear after the envelope. Headers can contain different types of application-specific information like security authentication or session management info.

3) Body – Sometimes referred to as the payload, the Body contains the actual message that shows the information for the recipient of the message.

10. Commonly used HTTP Methods

10.1 GET Method

The HTTP GET method is used to retrieve data. It is a read only call and data cannot be modified. If it is a successful request, without any errors, then a status of code of 200(OK) is received along with the data requested in the particular format which is generally JSON or XML.
GET method is considered a safe method to call as no corruption or modification of data will be done.

10.2 POST Method

The HTTP POST method is a write method. It is mainly used to create a new resource by providing the input in Json or XML format.
When a POST request is made, a new resource is created with the values provided along with it. If it is an error free call, the status returned will be 201.
POST methods are responsible for modification of data and should be carefully handled.
These methods are not idempotent and thus can result in errors or duplicate records if called more than one time.

10.3 PUT Method

The HTTP PUT method is more like an update command. It is used to change the value of any resource whose original value was something else.
PUT methods can also be used to create resources but only if the the request is from the client and not the server.
PUT methods are also not safe as these are responsible for resource modification but if we call the same PUT request again then no changes are made i.e it is idempotent.

10.4 DELETE Method

The HTTP DELETE method, as the name suggests, is used to delete a resource.
If the request is made successfully then the status code returned is 200 (OK). These are also idempotent as if a resource is deleted once it cannot affect it.

11. HTTP Response codes

The HTTP response codes are divided broadly in five categories :
• 1xx – Informational codes
• 2xx – Success codes
• 3xx – Redirect codes
• 4xx – Client error codes
• 5xx – Server error codes

Some of the most common HTTP response codes used with REST are as follows :

200 (OK): This code indicates that the request was made successfully.
201 (Created): This response code indicates that request was successful and a resource was created. It is used to confirm success of a PUT or POST request.
400 (Bad Request) : This code indicates that the data is in incorrect format.
401 (Unauthorized) : This code indicates an authentication error.
405 (Method Not Allowed) : This code indicates that HTTP method used is not supported for this resource.
409 (Conflict) : This code indicates that there is a conflict request to create the same resource twice.
404 (Not Found) :This code indicates that the required resource could not be found.
500 (Internal Server Error) : This code indicates that there is some error on Server side.

The Benefits of API Testing

Early Testing –
With API testing, once the logic is designed, tests can be built to validate the correctness in responses and data. We don’t have to wait for various teams to finish their work or for full applications to be built – test cases are isolated and ready to built immediately.
Easier Test Maintenance –
UIs are constantly changing and moving around based on how they are accessed – browsers, devices, screen orientation, etc. This creates a nightmare scenario where tests are being constantly rewritten to keep up with the actual code in production. API changes are much more controlled and infrequent – often times API definitions files like OpenAPI Spec can help make refactoring tests only a seconds of work.
Faster Time To Resolution –
When API tests fail, we know exactly where our system broke and where the defect can be found. This helps reduce time triaging bugs between builds, integrations, and even different team-members. The small, isolated footprint of an API test is perfect for faster MTTR stats, a valuable KPI for DevOps teams.
Speed and Coverage of Testing –
100 UI tests may take 10 hours to run whereas 100 API tests could be run in 1 hour too. That means we can find and fix more bugs in less time.

Popular API Testing tools

Having the right process, tool and solution for API test are critical for success. Below are the tools which are quite popular in API testing space:

Postman
SOAP UI
REST-Assured
Swagger.io
JMeter
Katalon Studio

I’ll give a list of API Testing tools in next article where we ‘ll see some different API Testing tools.

Security Testing Tutorial

What is Security Testing?

Why is Security Testing Important?

Types of Security Testing

Penetration Testing

Vulnerability Scanning

Security Auditing

Security Code Review

Threat Modeling

Security Testing Process

Planning

Scanning

Analysis

Reporting

Remediation

Security Testing Best Practices

Define a Security Testing Strategy

Understand Your Application

Create Test Cases

Use Security Testing Tools

Test for OWASP Top 10 Vulnerabilities

Conduct Regular Testing

How to perform security testing parallel in each stage of the software development life cycle(SDLC)

Example Test Scenarios for Security Testing

Security Testing Tools

Web Application Security Testing:

Network Security Testing:

Code Analysis and Static Analysis:

Mobile Application Security Testing:

Cloud Security Testing:

Database Security Testing:

Operating System and Server Security Testing:

Data Migration Testing | Test Strategy | Testing Phases | Challenges

In the last article on Data Migration, we learnt What is the Data Migration, Types of it and various strategies. Now it’s the time to learn about Data Migration Testing.

What is Migration Testing?

Why Migration Test?

As we know, the application migration to a new system could be for various reasons, system consolidation, obsolete technology, optimisation, or any other reasons.

Hence while the System in Use needs to be migrated to a new system, it is essential to ensure the below points:

Any kind of disruption/inconvenience caused to the user due to migration needs to be avoided or minimised like downtime, loss of data.

Need to ensure if the user can continue to use all the features of the application.

To ensure compatibility of the new/upgraded application with the hardware and software that the legacy application supports.

To ensure the existing functionalities works as in the legacy application seamlessly.

Critical Defects related to the data and data type need to be identified & fixed during testing.

To ensure no performance degradation post migration.

To ensure if the connection between servers, hardware, software and therefore the Data flow between different components remains intact.

Hence in order to ensure a smooth migration by eliminating the above defects, it is essential to carry out in-depth Migration Testing.

Importance of Data Migration Testing

There are some challenges that could arise during data migration testing, which is precisely why the testing itself is so crucial. Some of these challenges include:

Data corruption

Testing the mapping or path of data.

Unintended data modifications in the legacy system and in the data structures mapping.

By uncovering and resolving these issues during the testing phase can be priceless and a success.

Test Phases in Migration Testing

The different phases of Migration test need to be considered:

1. Pre-Migration Testing2. Data Cleansing3. Post Migration Testing

Phase 1 : Pre-Migration testing

Before any migration has taken place, the following testing actions should take place:

Ensure that the scope of the data is clearly understood what data has to be included, what data has to be excluded, which data needs.

Ensure that the destination as well as the load process is understood.

Perform data mapping between legacy and the new application.

Ensure that the data scheme is known including mandatory fields, field names, field types, data types etc. for both the original data source and the destination system.

Understand the data cleansing requirements.

Understand any interfacing with other systems or secondary suppliers of data into the system.

Tables in the legacy system are to be noted down and if any tables are dropped and added post migration needs to be verified.

Records in each table, views should be noted in the legacy application.

Prepare test cases, test scenarios, and use cases for new conditions in the new applications.

Execute a set of test cases, scenarios with a set of users and keep the results, logs stored. The same needs to be verified after Migration to ensure that legacy data and functionality are intact.

Count of the data and records should be noted down clearly, it needs to be verified after Migration for no loss of data.

Ensure the mapping to user interface is correct.

Ensure the mapping to business process is correct.

Test tool configuration and script generation using field details.

Identify data subset (sampling) details for testing.

Ensure all business cases, user stories, or use cases are understood.

Phase 2 : Data Cleansing

It is vitally important that data cleansing is performed on a system that houses and maintains data. The following data cleansing tasks should be planned:

Understand the error types: these may include blank fields, too-long data lengths, or bad characters.

Identify method of checking the data: either interrogation through database using SQL commands or the use of data migration tools.

Understand the data dependencies i.e. if changing the data in one table impacts the data in another linked table.

Fix the data issues.

Verify the count of the data.

1. Pre-Migration Testing
2. Data Cleansing
3. Post Migration Testing

Identify the appropriate Tool for Migration: While defining the strategy of this testing, automated or manual, identify the tools that are going to be used. e.g. Automated tool to compare source and destination data.
Identify the appropriate Test Environment for Migration: