Chapter 9 Historical Quotes Downloader
|Top Previous Next|
The Historical Quotes Downloader is a utility program that allows downloading free end-of-day market data from the Internet and saving it in files on your computer’s hard disk. The Downloader does not download intra-day historical quotes. Downloaded data files can later be loaded into Ward Systems Group’s ChaosHunter and NeuroShell Trader programs. A connection to the Internet is required to download data.
The Downloader can download daily market data from three different web sites: Yahoo! Finance, QuoteMedia, and Google Finance. We have included three different quote sources in the event one source is unavailable or has changed its historical data download link and stopped responding, you could use the remaining sources to get data.
All three quote sources provide historical daily data for major exchanges in the US and other countries around the world. Available instruments are stocks, mutual funds, ETFs, futures, indexes, and currencies.
In addition to historical quotes, both Yahoo! and QuoteMedia provide today’s prices delayed by 15-20 minutes depending on the exchange.
The Downloader does not care what type of instrument you request. All that is required is a valid symbol. The Downloader cannot do the symbol lookup for you. Instead, for a selected quote source, you must first figure out valid symbols by visiting the quote source web site and by doing the symbol lookup. For stocks, mutual funds and ETFs this should be simple enough. For futures and currency symbols this can be a research project in itself. Ward Systems Group, Inc. cannot provide help and technical support for symbol lookup.
Output File Formats
The Downloader saves data in two different file formats: ChaosHunter .bin format (available in ChaosHunter 3.0 or higher), and in NeuroShell Trader .wsg format.
The Downloader has several parameters that you need to set before requesting data from the web. Below is a screenshot of the program interface with the parameters described below:
Quote Source – a dropdown box that selects the web site from which to download data. Options are Yahoo! Finance (the default setting), QuoteMedia, and Google Finance.
Output Format – a dropdown box that selects the format in which downloaded data is saved onto the hard disk. Options are ChaosHunter .bin file (the default setting), and NeuroShell Trader .wsg file.
Output folder – a destination path on the hard disk where the Downloader saves data. Clicking on the small command button on the right brings up a standard Windows file dialog that allows you to select the destination folder.
If the default output folder does not exist, it is automatically created before data files are saved.
In the case of ChaosHunter, the default output folder is C:\ChaosHunter\DailyQuotes.
In the case of NeuroShell Trader .wsg format, the default output folder is C:\NeuroShell Trader 6\DailyQuotes. Map this folder in NeuroShell Trader using the Tools Menu, Data Sources, Directories and Files tab. It is advisable to download data when NeuroShell Trader is closed because NeuroShell Trader only analyzes data folders and discovers new symbols at program start up.
Start year – a text box that specifies the year from which to start downloading historical quotes. Enter the start year as a 4-digit number between 1900 and the current year. The program defaults to the year 2000 as the start year. Do not include the month and day in the start year (e.g., don’t enter a full date such as 1/1/2000 as the start year).
The end date for the download is not specified. The Downloader requests all data that is available through today.
Symbol List – a list of symbols to download. Use the Add, Edit, and Remove buttons below the list to manage the symbol list. The symbol list is saved as a text file named Symbols.txt in the same folder where the Downloader exe resides. Optionally, the list file can be managed and edited in Windows Notepad or WordPad (or any other text editor of your choice). Each separate line in the symbols.txt is a symbol. The Downloader reads the symbols.txt when the program starts and saves the list file when the program ends. Therefore, edit the list file only when the Downloader is not running.
Each symbol on the list has a check box at the left, thus a symbol can be either checked or unchecked. When the Download Checked button is pressed, only checked symbols are downloaded. The Download All button downloads all symbols regardless of their checked status. The Downloader does not remember which symbols are checked between program runs.
After download is complete, the symbol list displays (for each symbol) the status, the number of bars, start and end dates, and the file name under which the data is saved.
The status can have three different values: Success, Request Failed (check symbol), and No Data.
The Request Failed status usually indicates that the web site did not recognize the symbol. To resolve this, go to the Quote Source web site and search through their symbol lookup for the exact symbol for the instrument of interest.
The No Data status indicates that the symbol is recognized by the web site, but there are no historical quotes available for it. To resolve this, try a different Quote Source for the same instrument, but first verify that you have entered the correct symbol for the instrument on that particular quote source. Same instruments may be designated by different symbols on different quote sources.
Data Quality And Post-Processing
For all three quote sources the volume is frequently unavailable and is set to zero in the incoming data. No special adjustment is made for zero volume.
1. Yahoo! Finance
1.1. Yahoo! sends out raw data containing 7 columns: Date, Open, High, Low, Close, Volume, and Adjusted Close. The Open, High, Low, Close, and Volume data are unadjusted for dividends and splits, while the Adjusted Close is corrected for dividends and splits. Since unadjusted price data has large gaps in it due to splits, such data is not very good for modeling. Therefore, the Downloader adjusts raw data before it is written out into the file in the following manner:
-the Adjusted Close is copied as the Close column with no alterations.
-The Open, High, Low, and Volume columns get scaled using the ratio between the Adjusted Close and unadjusted Close.
1.2. Yahoo! sends currency data with only 2 digits after the decimal point that makes the currency data not very suitable for modeling purposes. We suggest that you use QuoteMedia for currencies.
2.1. The QuoteMedia stock data seems to be already adjusted for splits (not dividends). Since no adjustment is made to dividends, the data will be different than Yahoo!’s data that is adjusted for both splits and dividends.
2.2. QuoteMedia seems to be a better source for currencies because it sends prices with 4 digits after the decimal point.
3. Google Finance
3.1. The Google Finance stock data seems to be already adjusted for splits (but not for dividends). Since no adjustment is made for dividends, the data will be different than Yahoo!’s data which is adjusted for both splits and dividends.
3.2. Google sends the Open price for stocks as zero for all dates before the year 2000. The Downloader sets the zero prices to the Close price of the same bar.
3.3. Sometimes, Google data contain phantom historical bars for dates when markets were actually closed for major US holidays (e.g., New Year’s Day, Memorial Day, Independence Day, Labor Day, Thanksgiving, Christmas, etc.). Phantom bars have exactly the same OHLC prices that equal the previous bar’s close.
One can easily spot these phantom bars by looking at the volume which is zero for these days. The Downloader does not attempt to remove the phantom bars. There is no harm in having the phantom bars present in the data that are used to create models in either ChaosHunter or NeuroShell Trader since phantom bars are rare and will most likely not affect the model’s optimization process.
3.4. Sometimes, a trading day immediately preceding a major holiday (such days trade for shorter hours) has exactly the same OHLC prices, which is highly unlikely. The value of the OHL prices seems to be the close of the bar. Such days with equal OHLC prices have zero volume as well.
NeuroShell Trader® and ChaosHunter® are registered trademarks of Ward Systems Group, Inc.