Could you sum up some of the more significant drawbacks or problems that users have come across when backtesting ?

The Backscanner

Bug:

The annual return is overstated by a factor of  3.69 or 24/6.5.

Shortcomings:

No entry or exist @price

No money managment

No optimization of runs to
   o optimize parameters
   o easily conduct entry and exit studies

No IO capabilities

No multiple entries of same stock for scaling inot and out of positions

No easy way to clearly document the details of the backtest along with the performance results

Apparently no future for backtesting capability in the Woden product line.

Backtesting in general

1. It is easy to trick oneself into thinking excellent backtesting performance is the result of some significant truth...

When in fact either there is future information of some type skewing tthe results or the set of trades what over optimized to the point of curve fitting or cherry-picking. As a result the performance can no be matched in acutual trading. If the result seem to good to be true... they probably are.

2. Long backtest that show spectacular results are often the result of unrealistic trades. That is the size of the position toward the end of the backtest would be unfillable in a true market position. $200,000 dollars invested in a thinly trades $2 stock with a 100% return could not realistically be filled.

3.  The actual performance of the backtested strategy going forward will almost always be less tan the backtested results due to the nature of tuning the results and finding the sweet spots.

4. Survivorship bias and membership bias have a tendency to produce unrealistice results.
 Survivorship: Those stocks in the market todays that are being backtest against are a subset of the stocks that existed during the time that is being backtested.
 Membership: the stocks in the S&P 500 today are there as a result of performing well over time. So using the S&P 500 watchlist to backtest will typically produce better results that testing against 500 randomly selected stocks.

=====================================

Merits of Backtesting

All this said the backscanner can provide significant insight into the merits of a particular stategy, as well as adjusted versions of the strategy. This can provide confidence that the stategy has merit or strong confirmation that is does not.

---------------------------

One last point. The backscanner does provide the way to run a "randomized"  backtest over the testing period, typically with fixed length trade durations and can provide a benchmark to measure the stragey of interest against. If one tests a stratgey with a backtest performance return of x%, and the randomized test has a performance better than x%, then one would/should  strongly question the merit of the strategy.