Open topic with navigation

Testing Protected Applications

Any time you ship your application or make it available to customers and prospects, it is pivotal to test everything thoroughly. The licensing logic added to your application should be included with this testing to help prevent unforeseen issues from arising.  This topic will help guide you through this process, and may even serve as a helpful 'checklist' for you to print-out to keep track of what test has been done for new and updated applications.

Exercise Caution

Oftentimes, testing on computers that are used regularly and/or used for development can be less than ideal, can even result in unintended side effects. In general, we strongly advise you to avoid such problems by using virtualization software as described in the next section. A good example of where things can go awry when testing on development computers is the negative impacts that are possible from altering your system clock (which you might try to do when testing time-limited licenses). Some examples of how this could affect your development systems includes (but is not limited to):

• The licenses for software you have installed can begin to fail validation. This is because the licensed applications think so much time has passed that they must re-validate with the vendor's licensing servers again; however, validation with such servers is likely to fail because your system clock has been altered significantly. Visual Studio 2013 is one example which can be affected in this manner by altering your system clock, and this has the potential to be disruptive.
• Anti-virus and security tools may begin showing errors about being out-of-date.
• Automatic appointment/task reminders (in applications such as Outlook) could be shown prematurely. If, for example, these are dismissed during testing, the reminders will not be shown again when originally intended because they were already dismissed.
• When committing/checking-in source code changes into a repository, the dates on the commits could potentially be wrong. Similar issues could occur with any software that utilizes the system date.

If you cannot avoid these issues by using virtualization software, then the next best thing to do is to have a dedicated test machine which is backed-up with a complete image/snap-shot regularly.

General Testing Practices and Tools

As with any task, it is important to ensure you have the right set of tools to help ensure the task is completed well and efficiently. Here are some types of tools that we can recommend for your consideration.

Virtualization Software

Although this is not required, using virtualization software to run virtual machines is very helpful for a wide variety of reasons.

• It is possible for you to establish different virtual machines for testing different operating systems and versions thereof.  For example, you could have separate virtual machines configured for testing Windows Vista, Windows Server 2008, Windows 7, Windows Server 2008 R2, Windows 8, etc... Even amongst these, you could have separate virtual machines for the 32 bit and 64 bit builds of each applicable version of Windows.
• Most virtualization programs support saving "snapshots" of each virtual machine guest, which allow you to restore the virtual machine guest to the exact state it was in when the snapshot was taken. For example, if you tested installing an application on a system for the first time ever, you can revert to the snapshot when done if you need to run the test again. So if you had an issue with performing a fresh installation of your application that could not be reproduced when re-installing it, you could reproduce the problem by simply reverting back to the snapshot and starting over.
• The virtual machine guests and snapshots can be configured in a variety of different ways to cover different testing scenarios.  For example, if you sell an application that also has related developer components that must be activated separately, you could test installation and usage with and without the relevant developer tools and dependencies to ensure the smoothest experience possible for all of your users.

Keep in mind that if your licensing requirements call for the detection of virtual machine guest environments to alter the way your licensing logic behaves (or even to prevent the application from running in these environments entirely), you may need to make additional considerations and adjustments for your testing procedures. These adjustments can include creating special test builds which are never publicly released, and could include testing on physical computers in place of or in addition to testing in virtual machine guests.

Build and Test Automation Scripts

There are many free and commercial tools available for automating your build and testing processes.  Some free tools include, but are not limited to, Power Shell scripts or batch files, shell scripts, home-grown scripts and applications you might already have in place. Having scripts related to building and testing helps avoid missing crucial steps to building applications properly, and also helps automate any mundane building and testing tasks. Additionally, unit testing can also help identify potential issues during development, and is a particularly convenient way of testing all possible execution paths in your code. Most integrated development environments (IDEs) have built-in features for assisting with unit testing, and many also have various code analysis features that can also help identify areas of code that could be problematic or be adjusted to follow best practices.

Establish a Testing Checklist

Since every application is different, each will have a different set of tests that should be run before releasing a new build to customers and prospects.  Below is a list designed to help you get started on a checklist for testing the licensing features (and potentially other features) in your applications.

Verify your Installer

Installing

• Is the application, all of its dependencies, and any required files and permissions installed correctly?  See the PLUSManaged deployment or PLUSNative deployment section for details on deploying protected applications.
• If your application should be activated once for each computer/device, does the installer prompt to obtain elevated privileges?  This is required to install licenses shared with all users on a single computer, and is required to avoid problems with User Account Control (UAC) features on Windows Vista and later. This is also described in the PLUSManaged deployment or PLUSNative deployment sections.
• If your application should be activated once for every user on a computer/device, does the installer use user-centric locations?  If Windows displays a prompt asking to confirm if the application installed correctly, this might indicate that the installer attempted to use global locations instead (which could indicate potential for UAC related issues).
• After installation, does the application start in the expected state?  For example, if it should be a 30 day trial, does it start as such?

Upgrades

• Does the application, its dependencies, and required files get updated as appropriate?
• If the upgrade or update is free for customers who have already purchased, downloaded, installed, and activated, confirm that reactivation is not required after upgrading.
• If the upgrade is not free for customers who have already purchased, downloaded, installed, and activated, confirm that a new activation is required by your application after upgrading.
• After installing the upgrade, does the application start in the expected state? This can vary based on whether the upgrade is free or not, as described above.

Uninstalling

• Make sure the uninstaller removes the application, its dependencies, and files, as appropriate.
• If you are using a writable license file with aliases, make sure the aliases remain on the system after uninstalling.  This is important so that uninstalling and reinstalling a trial does not afford users a fresh trial period, for example.

Activation and Electronic License Management

When testing activation and Electronic License Management (ELM) features in your application, it is important to test each possible means of performing the tasks that users are allowed/able to perform.

Using SOLO Server

You should perform the tests below with a test system that has a direct Internet connection if you allow activating online. If you allow activation through another system's Internet connection, make sure you test with manual request files while the test system lacks Internet connectivity. Furthermore, if your application supports several different types of licenses (such as full/non-expiring, time-limited, network floating, etc...), make sure you also run the tests below for each type of license.

• Try activating with a bad License ID and Password and make sure it fails, and make sure you receive a result code of 5008 from SOLO Server.
• Try activating with a License ID for a different product, and make sure you receive a result code of 5010 from SOLO Server.
• Activate with a valid License ID and Password for the correct product. Make sure it goes through successfully.
• Refresh the license.
• Deactivate the license from your application (if allowed).
• Reactivate the license using the same License ID and password.  If SOLO Server is configured to allow reactivation on the same computer, you should receive the same Installation ID. Otherwise, you should receive a new Installation ID, or a result code of 5013 if no activations are remaining.
• Use the SOLO Server management interface to revoke/ban the Installation ID, and refresh the license to ensure your application revokes its license properly.
• Reactivate the license using the same License ID and password, use the SOLO Server management interface to revoke/ban the Installation ID, and deactivate the license from your application (if allowed) again. SOLO Server should return result code 5016, and your application should typically revoke its license in this circumstance as well.

Using Trigger Codes

Activation is also possible using a numeric challenge-response mechanism called trigger codes.  Trigger codes afford your customers the convenience of being able to activate systems which lack Internet connectivity (especially systems in remote locations) by manually exchanging numeric codes with you or your customer service representatives via phone or fax.

By definition, a trigger code is essentially just an encrypted value which gets decoded to a numeric value between 1 and 50 (which is the trigger code number). When processing the activation codes, the resulting trigger code number is used to then identify which actions should be taken in your program to alter the state of the application's license. The Protection PLUS 5 SDK sample applications include some basic trigger code implementations for the most common types of licenses. Make sure you run the following tests:

• Test activation with each trigger code number supported by your application. Make sure the application activates without issue, and make sure the license is altered as intended.
• If you allow activation via trigger codes while also allowing for activation with SOLO Server, you will need to test how things behave when mixing both together. For example, if you allowed users to activate time-limited licenses and non-expiring licenses:
  • Test activating a time-limited license with SOLO Server. Then activate it as a non-expiring license using a trigger code, and then refresh the license with SOLO Server. Make sure that refreshing the license does not revert it to its time-limited state. If it does, you may want to consider removing the Installation ID from the license file when the type of license (i.e. changing from time-limited to non-expiring in this example) changes due to a trigger code.
  • If trigger codes are used to activate individual application features and/or modules in a disconnected state, it will be important to ensure a refresh does not overwrite this data with old or empty data.  The application should be designed so that the trigger codes update user defined fields in the license file. Prior to issuing such a trigger code to a customer or prospect, the SOLO Server license should be updated so that its user-defined fields will match what is expected to be in the license file as a result of issuing trigger codes.

Copy Protection

Your application can identify a system which has been activated to ensure its license is only used on the system on which it was activated.  Each application can use a different combination of algorithms and requirements for this purpose, so here are some generalized tests for you to consider for each identifier algorithm being used:

• Install the application on two systems. Activate on one system, copy the activated license file to a second system.  Make sure the application does not run. For applications designed for home or non-business use, this is generally a sufficient test. However, if your application is designed to be installed in large corporate environments, academic institutions, etc..., you may want to consider testing on separate computers which share very similar hardware.
• Install and activate the application on a system, and test performing typical hardware changes/upgrades to see the impact for each identifier algorithm being used. So for example, if you are using the NicIdentifierAlgorithm, you may want to test adding and removing a USB wireless network card in a laptop, if that is a reasonable scenario to expect for your application.

Time-Limited Licenses

If your application uses time-limited licenses (which are licenses that are only valid for a limited amount of time), then you should test your application's behavior thoroughly.

If your application supports multiple types of time-limited licenses (such as an evaluation and an activated, lease or subscription style license), run the tests for each type of license. Before you begin your testing, make sure your test system is not configured to synchronize with network/Internet time.  In Windows, you can change this setting in the Internet Time tab located in the same dialog where you can adjust your system's date/time.

• Start the application, and activate it if and as appropriate (activation is often not required for evaluations).
• If your application attempts to check its status or refresh its license with SOLO Server, push the test system's date forward to the date when you expect it to start the attempts.  It should fail with a result code of 5022, as the system clock will have a differ from SOLO Server's clock by more than 24 hours. Try unplugging/disconnecting the system's network adapter on the test system , and start the application again. If the refresh is not required, the second attempt should fail silently.
• If your application requires checking its status or refreshing its license with SOLO Server, push the test system's date forward to the date when you expect the requirement to be effective. Make sure the application does not run.
• Leave the network adapter disconnected, and back-date the system's clock to the correct date/time. Without Internet connectivity, alias validation should fail and your application should treat the license as being invalid or expired. Reconnect the network adapter, and test to ensure refreshing the license online (which should occur automatically) restores the license to a functioning state.

Internet Connectivity

If your application requires checking its status or refreshing its license with SOLO Server, you will need to make sure you test this functionality with and without Internet connectivity. Here are some tests you can run:

• If your application has an option, menu, or button that can cause a status check or license refresh to occur, start the application and test this functionality with and without Internet connectivity.
• If your application is configured to begin checking or refreshing its license after a period of time, push the test system's clock ahead to the date you expect checking to begin, and make sure SOLO Server returns 5022 (which will indicate the fact that SOLO Server's clock differs from the test system's clock by more than 24 hours). Also test with no Internet connectivity, and make sure the application continues without showing any errors regarding the failed attempt.
• If your application is configured to require checking or refreshing its license after a period of time, push the test system's clock ahead to the date you expect checking to begin. Otherwise, if your application always requires checking or refreshing its license, you should not need to later the test system's clock. Make sure SOLO Server returns 5022 (which will indicate the fact that SOLO Server's clock differs from the test system's clock by more than 24 hours). Also test with no Internet connectivity, and make sure the application shows an error and does not run.

Build Configurations

Sometimes it is necessary to use multiple build configurations of an application to produce different builds or releases for various reasons.  Reasons could include things like branding changes, using separate builds to omit certain features that are not easily enforced through licensing code alone (i.e. an evaluation build is sometimes necessary to omit unprotected content), etc... All tests should ideally be performed against all possible build configurations, especially when code is conditionally compiled for different build configurations (typically done via preprocessor directives).

Supported Environments

All tests should be repeated for each platform supported. So for example, if your application is supported on Windows 7 and later, you would typically test your application on the 32 bit and 64 bit versions of Windows 7, Windows Server 2008 R2, Windows 8, and Windows 8 Server.

Additionally, if your application supports multiple languages, all tests should be repeated with the typical regional settings expected with each language supported.

Load Testing

Many standard desktop applications exhibit their load on systems during normal use and testing.  However, if your application is expected to be run in an environment using terminal services or Citrix, or if your application includes a web application, service, or any multithreaded functionality, it is very important to test your applications performance under heavy use.  For example, with web applications, you would want to avoid checking the license status or refreshing the license file every time a page is hit.  Likewise, if your application were to be used by hundreds or thousands of users on a single machine via terminal services or Citrix, frequent license validation (especially validation with SOLO Server) could congest system and network resources. Furthermore, if your application supports network floating licensing via semaphore files, it is also very important to test load test this in a realistic simulation before deploying the application to a site.

Regression Testing

As you apply changes, updates, and fixes through its lifecycle, it is important to perform regression tests to ensure that recent changes do not introduce new issues. Although it may not be practical to test everything each time a new build is created, it is best to automate as many tests as possible (often done via unit tests) to help prevent the introduction of new issues.