testing tools https://www.softwaretestingportal.com Let's learn something new.....together Mon, 21 Sep 2020 11:49:04 +0000 en-US hourly 1 https://wordpress.org/?v=7.0.7 https://www.softwaretestingportal.com/wp-content/uploads/2018/09/cropped-Logo-again-middle-1-1-32x32.jpg testing tools https://www.softwaretestingportal.com 32 32 Data Migration – Plan, Strategy, Process and more https://www.softwaretestingportal.com/data-migration/ Mon, 10 Aug 2020 10:31:18 +0000 http://www.softwaretestingportal.com/?p=1418
 
Businesses are spending billions of dollars in migrating data in migrating systems, yet many of the new systems fail to meet expectations.
In fact More than 50% of data migration projects will exceed budget and/or failing the business due to flawed strategy and execution, according to Gartner. Nevertheless, data migrations are a necessity and fact of IT life.
So let’s understand this complex though inevitable process at a glance.

What's Data Migration!

Data Migration is a process that includes moving data between formats, computer systems, and storage types.
We can also say that it is the process of selecting, preparing, extracting, and transforming data and permanently transferring it from one computer storage system to another.
Additionally, the validation of migrated data for completeness and the decommissioning of legacy data storage are considered part of the entire data migration process.

Why Data Migration!

 
  • Upgrading Computer / Legacy systems
  • Changing Computer / Legacy systems
  • Merging systems i.e. Data Integration
There are many reasons your enterprise might need to undertake a data migration project. For example, you might be replacing servers or storage devices or consolidating or decommissioning data center.
Data migration is also an essential step in the overall process of migrating on-premises IT infrastructure to a cloud computing environment.
Whether you’re moving to a public cloud, private cloud, hybrid cloud, or multi cloud environment, you’ll need to find a secure, cost-effective, and efficient method of migrating your data to its new storage location.
Improved technology often means business is thriving and is improving alongside its competitors. That said, the act of migrating data can leave previous information vulnerable. If critical data is lost, the business could be left at a standstill.

And What’s not the Data Migration!

 
Note that data migration is not the same thing as data conversion or data integration:

Data Migration

Data Conversion

Data Integration

 
Moving data between storage devices, locations, or systems. Includes subsets like quality assurance, cleansing, validation, and profiling.
 
Transforms data from a legacy application to an updated or new application. The process is ETL: extract, transform, load.
 
Combines stored data residing in different systems to create a unified view and global analytics.

6 Types of Data Migration

 
 
Data is stored on various media in files or databases, and is generated and consumed by software applications, which in turn support business processes. The need to transfer and convert data can be driven by multiple business requirements, and the approach taken to the migration depends on those requirements. The major migration categories are as follows:

Storage migration

Data center migration

Application migration

A business may choose to rationalise the physical media to take advantage of more efficient storage technologies.This will result in having to move physical blocks of data from one tape or disk to another, often using virtualisation techniques.
Migrating an entire data centre environment to the cloud or another location is a large-scale, comprehensive process. Completing such a migration project successfully—with minimal downtime or disruption to operations—requires careful planning and coordination.
Changing application vendor—for instance a new CRM or ERP platform—will inevitably involve substantial transformation as almost every application or suite operates on its own specific data model.

Database migration

Business process migration

Data migration to the cloud

  • It may be necessary to move from one database vendor to another, or to upgrade the version of database software being used.
  • Some modern applications are written to be almost entirely agnostic to the database technology, so a change from Sybase or SQL Server to Oracle should only require a testing cycle.
  • Database migration is an example of specialised workload migration.
  • Many public and private cloud providers offer tools that can facilitate or automate parts of the database migration process to ensure that your database remains secure throughout the transfer and that no data loss or corruption occurs. 
 
  • Business processes operate through a combination of human and application systems actions.
  • When these actions change they require the movement of database or application data.
  • Examples of such migration drivers are mergers and acquisitions, business optimisation, and reorganisation to enter new markets or respond to competitive threat.
  • The last two categories directly affect the operational users of processes and applications.
  • A highly adaptive approach,  a business-oriented audit and clear visibility for stakeholders are the key requirements in such migrations.
 
 
  •  
  • Today, businesses generate ever-growing amounts of data and face increasingly urgent pressure to maximise the value they extract from it. 
  • Therefore, many organisations are choosing to move workloads to the cloud for a cost-effective and best-performing IT environment.
  • There are several options for transferring data from a local data centre to the cloud, broadly, they fall into following two categories.

    1. Online migration, in which data moves across the Internet or a private or dedicated WAN connection.

    2. Offline migration, in which data is transferred via a storage appliance that’s physically shipped between data centre and the target cloud storage location.

9 Steps for Successful Data Migration

1. Design a Strategy

  •  
  • Selecting a data migration strategy depends on the need for data migration. Is it because the organization is consolidating systems following an acquisition or a merger or is it because of data overload?
  • The design phase is where organisations define the type of migration — big bang or trickle. This also involves drawing out the technical architecture of the solution and detailing the migration processes.
  • Then you needs to decide whether to go for data migration services or one of the ETL tools to facilitate data migration.
  • These tools will identify the source and destination systems, generate workflows, and migrate the content. Alternatively, a code-free data migration solution can give you the freedom to carry out the migration at your convenience, speeds up the project, and reduces cost and risks.

2. Assess the Data source and Analyse

Before migrating data, you must know what you’re migrating, as well as how it fits within the target system. 
So the next step is to analyse enterprise data. Answering the following questions can help with this assessment:
  • How much data is pulling over?
  • What that data looks like?
  • How much of the enterprise data requires migration?
  • Is the data structured or unstructured?
  • Is data redundancy going to be an issue?
  • What needs to migrate over, what can be left behind
  • How much of the data is ROT (redundant, obsolete, or trivial)?
  • Is the information old or recent?

3. Build a Migration Solution

Since, data migration is a one-time and large activity, it’s crucial to get it right.
A common tactic is to break the data into subsets and build out one category at a time, followed by a test. For particularly large migration projects, it might make sense to build and test in parallel.

4. Collect and Cleanse Data

This step involves removing data which is not required, compressing the remaining content, and converting data into the desired format.
Manual execution of these tasks can be time and resource intensive for large datasets. Therefore, automating these tasks can make the migration process more efficient.

5. Sort Data

Once you have profiled the data into a high quality and usable format, the next step is to categorise it according to the migration requirements.
It makes easier to route data into the right buckets. For example in Banking, data can be organised based upon the customer types, product type, portfolio types, account types and any other criteria.

6. Validate Data

This step involves reviewing the execution process. Assess the data rules and check if they are working the way they are supposed to, and watch for any exceptions in your dataflow.
After sorting the data, apply the data quality rules to determine the errors based on defined rules and to identify and eliminate the records in which the field ‘Quantity’ has a zero value.

7. Migrate

This is the final step in which everything falls into place. The steps mentioned above will give you well-organised and clean datasets.
Now all we have to do is migrate them from one system to another. Once the data is migrated to the target system, which can be a database or a data repository.

8. Conduct a Live Test

The testing process isn’t over after testing the code during the build phase.
It’s important to test the data migration design with real data to ensure the accuracy of the implementation and completeness of the application.

9. Audit

Once the implementation has gone live, set up a system to audit the data in order to ensure the accuracy of the migration.

Data Migration Strategies

“Big Bang” Migration

“Trickle” Migration

 
 
In a big bang data migration, as the name suggests, full transfer is completed within a limited window of time.
Live systems experience downtime while data goes through ETL processing and transitions to the new database.
It all happens in one time-boxed event, requiring relatively little time to complete. The pressure can be intense, as the business operates with one of its resources offline. This risks a compromised implementation.
 
 
Trickle migrations, in contrast, complete the migration process in phases.
During implementation, the old system and the new are run in parallel, which eliminates downtime or operational interruptions.
Compared to the big bang approach, trickle implementation can be fairly complex in design. However, usually reduces risks.

Advantages of big bang migration

Advantages of trickle migration

  • Less costly
  • Usually less complex
  • All changes happen one time only, in a relatively short space of time.
  • Less prone to expensive surprises.
  • Zero downtime required – as the migration is incremental
  • Easy rollback – If a single phase fails, it’s only that phase that needs to be rolled back and repeated

Disadvantages of a big bang

Disadvantages of a trickle

 
  •  
  • A high risk of expensive failure as problems may be discovered after complete migration.
  • If the migration fails, a complete roll back is required.
  • Requires downtime.
 
  •  
  • More expensive.
  • It requires to maintain multiple live environments

Risks and challenges

1. Data Loss

2. Security

3. Compatibility

During the data migration process, data loss can occur.
On a small scale, this may not be a problem as there is less data or easy to restore the missing data or backup. However this is a major challenge in large scale migration programs.
Organisations need to make sure that all data is securely encrypted before migration.
For offline migrations that involve shipping data storage devices, it’s important to verify security of the shipper’s freight and logistics services.
Compatibility issues are also possible in data transfer, such as changed operating systems and unexpected file formats.
Although the data is not actually lost, however, business cannot access it in the target system.

4. Long transfer times

5. Unexpected Higher Costs

 
It can be challenging to predict  exact online transfer times.
Network impediments may restrict connection speeds or hardware restrictions might throttle the amount of data.
Unexpected costs often result from improper planning. For example, delays in online transfers additional charges.
Also, Keeping Vendor-provided storage appliances for longer than the agreed times can also raise the expanses.

Best Practices

1. Understand the data usage

2. Assess the environments

 
 
  • Who uses the data,
  • Who will use it in the future,
  • How will it be used? 
  • Formatting requirements ?
    Be sure to gather the requirements from relevant stakeholders and business units throughout the migration process.
 
  • Will the same operating system be running in source and target environments?
  • Will database schema or other formatting need to change?
  • Are there any issues that need to be addressed pre-migration? 

3. Verify business requirements

4. Back up the data

 
  • What kind of migration timeline is necessary?
  • If a data centre is being decommissioned, when will its lease expire?
  • What types of data security must bee maintained throughout the migration process?
  • Is any data loss or corruption tolerable, and if so, how much?
  • How would unexpected delays affect the business?
  • Make sure there are backup resources and that they’ve been tested before proceeding.
  • One of the good practice is to create backup images that can be immediately restored to the original system.

5. Stick to the strategy

6. Test, test, test

 
  • The migration process can be complicated at times, so be ready for it and stick to the plan.
  •  
  • Make sure everything’s where it should be, create automatic retention policy, clean up the data and double check permissions.
  • During the planning and design phases, and throughout implementation and maintenance, test the data migration to make sure to achieve the desired outcome.

Data Migration Tools

On-premise data migration tools

Open source data migration tools

Cloud-based data migration tools

  • On-premise solutions are designed to migrate data between two or more servers or databases within a large or medium enterprise/network, without moving data to the cloud.
  • These solutions are optimal if you are performing tasks like changing data warehouses or moving the location of your primary data store.
  • Some companies prefer on-premise solutions due to security restrictions.
  • Open source is software you can use, modify, and share because its design is publicly accessible.
  • Typically, open source solutions are free or lower in cost than commercial alternatives.
  • Open source data migration tools can be a practical option for migrating your data if the project is not too large or complex.
 
  •  
  • Cloud-based data migration solutions are the latest generation and are designed to move data to the cloud.
  • Cloud-based solutions are optimal if you are already storing your data in the cloud or if you intend to move your data to the cloud.
  • Many organisations are now going for cloud based solutions considering cost efficiencies and enhanced security in moving data from on-premise to the cloud.
  • Moreover, cloud-based data migration tools tend to be flexible about the types of data they can handle.
 
 
Today, there are plenty of tools to facilitate enterprise data migrations. Generally, Data migration strategy  determines which tools work best for the specific project.
Some popular Data Migration tools include the following:

• Rsync for host-based file-level migration
• Veeam to speed and simplify the process of migrating VM-based workloads.
• EMC Rainfinity for network-based file-level migration.
• Zerto offers a converged disaster recovery, backup, and workload mobility platform.
• HDS Universal Replicator for array-based block-level migration
• Rclone, an open source command line utility, used to migrate data to and from cloud object storage.
• Brocade DMM for network-based block-level migration
• Cyberduck is an open source FTP and SFTP client, used to transfer individual files or file volumes.
• Volume managers for host-based block-level migration
• Windows 10 Migration Accelerator
• Zinstall
• Acronis True Image
• Minitool Partition Wizard Free
• Laplink PCmover
• Samsung Data Migration Tool
• AOMEI Backupper Standard
• EaseUS Todo Backup/PC Trans Free
• Paragon Drive Copy Professional
• Samsung Smart Switch

Conclusion

Whatever the reason for the data migration, its ultimate aim should be to improve corporate performance and deliver competitive advantage.
To succeed, data migrations must be given the attention they deserve, rather than simply being considered part of a larger underlying project.
Lacking this and without proper planning, there is a high risk that the project will go over budget, exceed the allotted time, or even fail completely.
Following a structured methodology will reduce the pain of managing a complex data migration, whereas the correct choice of technology will lead to a successful outcome.
]]> 20 API Testing Tools – Beyond SOAP UI and Postman https://www.softwaretestingportal.com/api-testing-tools/ https://www.softwaretestingportal.com/api-testing-tools/#comments Fri, 10 Apr 2020 01:36:06 +0000 http://www.softwaretestingportal.com/?p=992

We all know about API testing well. In my previous article – API testing, Key Terminologies and more……we learnt about the basics of API testing. Let’s now take a look on some of the API Testing tools.

To learn something new, I haven’t considered the leaders like SOAP UI, Postman, JMeter. Rather I have explored some tools which may be lesser known but have more or less same features and give a solid performance therefore worth giving a shot.

#1 ACCELQ

  • Zero code API Test Automation on Cloud.
  • Support for leading technologies.
  • API and UI Test Automation in the same simplified flow.
  • CI Driven regression suite executions.
  • Dynamic Environment management.
  • Directly correlate Business process and corresponding API for complete coverage.
  • API Test Case Management, Test Planning, Execution and tracking governance.
  • Trial – 14 days free trial

#2 API Fortress

  • API Fortress is a REST & SOAP API, Web Services, and Microservices testing platform.
     Out-of-the-Box Integrations.
  • Deploy On-Premises or in the Cloud.
  • Quickly Generate and Edit Functional Tests
    Detailed Reporting.
  • 30 days free trial

#3 API Science

  • API Science is an advanced API Monitoring service.
  • Monitor own APIs
  • Multi-step REST transactions
  • Advanced tests & validations
  • Get alerted on any issue
  • Run tests at any time
  • Monitor your own API & 3rd party APIs
  • Customized reporting
  • Intelligent API diagnostics
  • Collaborate using
  • team plans

#4 Apigee

  • Easily create API proxies and visually configure or code API policies as steps in the API flow.
  • Enforce consistent security best practices and governance policies across all APIs.
  • Publish APIs by providing reference documentation, manage the audience for an API from portal.
  • Analyze APIs, API traffic data, investigate traffic spikes, and trace live API calls.
  • Monitor APIs to ensure API availability.
    Publish APIs to a customizable portal, so that developers can easily explore, test, get API keys, and innovate fast.
  • 60 days free trial.

#5 Assertible

  • Can automatically sync API tests with the latest changes in the specification.
  • Integrates with the tools e.g. push code to GitHub or send alerts to Slack if failures happen.
  • Easily create tests for web service.
  • Continuously test multiple deployment and environment.
  • Set up scheduled monitoring to continuously test the services.
  • Run the same tests on multiple environments.
    Automatically execute post-deploy smoke tests.
  • HTTP assertions allow you to create domain specific test.

#6 Citrus Framework

  • Citrus is open source and licensed under Apache License 2.0.
  • Test automation by creating repeatable integration tests within continuous build.
  • Test complex messaging scenarios with step by step sequence of operations.
  • Use advanced logic in test cases including loops, delays, retries, parallel sections.
  • Json message payload validation.
  • JsonPath element validation.
  • Groovy Json validation.
    Simple test creation.
  • IDE support – execute tests as TestNG/JUnit tests from Eclipse, IntelliJ IDEA or NetBeans.

#7 HTTPMASTER

  • Development and test tool for REST web services and API applications.
  • Can display and validate the most common REST formats; XML, JSON, and HTML
  • Broad set of http properties.
  • Response data validation
    Simulate variations of input data or create global values for reuse across multiple API calls.
  • Enables to define chaining items of different types.
  • Support for data upload which enables users to post plain text data, contents of file, or mix of plain text data and disk files.
  • Free 30-day trial

#8 Insomnia

  • Free and open source on Mac, Windows, and Linux.
  • Reuse API keys or session IDs. Define environment variables globally or switch between sub-environments.
  • Generate http code for over thirty language libraries, including Curl, NodeJS, Go, Swift, Python, Java, C.
  • Free version available.

#9 Mockbin

  • Mockbin allows you to generate custom endpoints to test, mock, and track HTTP requests & responses between libraries, sockets and APIs.
  • Mock custom endpoints using any HTTP Archive (HAR) response object.
    Test using your preferred language.
  • Mockbin accepts all standard Methods and allows method overriding in addition to Get and Post.
  • Mockbin supports output in JSON, YAML and XML, as well as an HTML.
  • Log and inspect incoming calls to your custom endpoints.

#10 Ping API

  • Ping-API is API testing allows to write test script in JavaScript and CoffeeScript.
  • It allows inspecting HTTP API call with a complete request and response data.
  • Can schedule test in every minutes or hour
    Support for writing script to set request headers, body and URL parameters.
  • It also supports validation of response headers and body.
  • Easy to generate test script with the input of API parameters.
  • Free for limited projects, varied pricing afterwards.

#11 Postwoman


Postwoman is an open-source API request builder. It helps you create your requests faster, saving you precious time on your development.

It also helps to create the requests faster, saving precious time on overall development.

  • Free, Open Sourced
  • Runs online
  • Self Hosted
  • Multi-platform & multi-device support
  • Progressive Web App
  • WebSocket testing
  • Customizations allowed
  • Pre-Request Scripts and environment variables setup
  • Docker support

#12 pyresttest

  • A REST testing and API microbenchmarking tool.
  • Tests are defined in basic YAML or JSON config files, no code needed.
  • Minimal dependencies (pycurl, pyyaml, optionally future), making it easy to deploy on-server for smoketests/healthchecks.
  • Supports generate/extract/validate mechanisms to create full test scenarios.
  • Returns exit codes on failure, to slot into automated configuration management/orchestration tools (also supplies parseable logs).
  • Logic is written and extensible in Python.
  • Return exit codes for failed results.
  • The building of test scenarios with generate/extract/validates mechanisms.
  • Easy deployment on the server.

#13 ReadyAPI

ReadyAPI is an integrated suite of API testing tools that includes:

  • SoapUI Pro (functional testing)
  • LoadUI Pro (performance testing)
  • ServiceV Pro (service virtualization)
  • Metrics & Data In One Place.
  • Create and maintain all the information relating to projects including test cases, data sources, and object components in one place for re-usability and management.
  • The entire ReadyAPI suite uses a central, standardized reporting function to support the measuring and displaying of data and metrics in real time.
  • Continuous Integration with ReadyAPI.
  • ReadyAPI provides many options for scripting to create functional, load, or security tests, using either Groovy or JavaScript.
  • With API Discovery, ReadyAPI will listen to live traffic of an API, recording responses and different interactions captured during the API activity.

#14 RestSharp

  • RestSharp is a popular HTTP client library for .NET.
    Featuring automatic serialization and deserialization, request and response type detection, variety of authentications.
  • JSON, XML and custom serialization and deserialization.
    Basic, OAuth 1, OAuth 2, JWT, NTLM are supported.
    HTTP call can be customized.
  • Send objects as the request body in JSON or XML, or as a form.

#15 SOAPSonar

  • SOAPSonar provides automated testing of each API independent of client, device or Consumer.
  • Automatically generating a UI to provide immediate feedback to developers, testers.
  • Validates each API independently, shortens the critical path and supports Agile development.
  • Fully automated, requires no scripting or maintenance thus reducing time and skill-set required.
  • Integration with variety test management software hence provides detailed progress reports.
  • Regression testing can be automated, providing early identification of changes to the API or versions.
  • Support for message and protocol layer identity standards allows for cloud testing.

#16 SOAtest

  • Broadest support for over 100 message formats and protocols.
  • With visual drag-and-drop tooling, users can create the most complex test scenarios without having to write a single line of code.
  • Test flow logic, complex assertions, looping, data driving, and keyword association, such as BDD with Cucumber, can all be easily built with minimal technical experience.
  • With its SMART API Test Generator plugin for Chrome, SOAtest monitors activity in your web UI from manual or exploratory testing and converts the behind-the-scenes API calls into meaningful API test scenarios.
  • Integration with existing CI/CD systems (i.e. Jenkins, TeamCity, Bamboo, Azure DevOps).
  • Collects REST and SOAP API coverage as part of test execution, reporting the test case coverage, missing or incomplete.

#17 TestMace

  • Creating and testing complex scenarios quickly.
  • Creating tests without actual programming.
  • Powerful autocomplete feature and highlighting of current variables values, functions etc.
  • A well-organized project structure and human-readable file format, which allow to store your project in the version control system repo and review tests and scenarios.
  • Advanced JS support in the built-in code editor, including the autocomplete feature and static analyzer.

#18 vRest

  • Record test cases by manually using your application once.
  • vrunner lets you execute your test cases just by running a command on your terminal.
  • Integration with Continuous Integration servers like Jenkins, TeamCity, Visual Studio Team Services etc. into vREST.
  • Automatically log issues into bug tracking systems like JIRA, Bugzilla, etc and notify team using Slack.
  • vREST lets you write test cases to validate your web/mobile application.
  • Supports Data driven testing. vREST lets you write test cases to validate your web/mobile application.
  • Schedule your test cases  using cronjob utility (in Mac or Linux) or Task Scheduler (in Windows).

#19 WebInject

  • WebInject is a free tool for automated testing of web applications and web services.
  • It can be used to test individual system components that have HTTP interfaces (JSP, ASP, CGI, PHP, AJAX, Servlets, HTML Forms, XML/SOAP Web Services, REST, etc).
  • Offers real-time results display and may also be used for monitoring system response times.
  • WebInject uses an XML API for defining and loading test cases. You can use WebInject without ever seeing its internal implementation.
  • Result reports are generated in HTML (for viewing) and XML 

#20 Paw

  • Paw is a full-featured HTTP client that lets you test and describe the APIs you build or consume.
  • It has a native macOS interface to compose requests, inspect server responses, generate client code and export API definitions.
  • Easily craft your HTTP requests, try and archive them.
  • Setup HTTP Headers, URL parameters, form-encoded POST key-values or text.
  • Get easy-to-read responses with well formatted headers and body.
  • Sync your projects with your team through Paw for Teams.
  • Simple request chaining (right-click in the UI, no code or globals).
  • Great workflow for when writing a fast-changing API

Conclusion

This list can go on and on however we have to stop somewhere. The above list is an effort to explore some different API testing tools which may not be the most popular ones but work well and perform effectively.

Though all API test tools listed above work well and are great options.however, there is no one-size-fit-all tool. There is no perfect tool for all organisations: Every organization has different requirements. 

So the key is – carefully considering the current and future project’s requirements and pros and cons of each solution.

]]> https://www.softwaretestingportal.com/api-testing-tools/feed/ 1